arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.14743cs.CLcs.LG
收藏

分布偏移下奖励模型准确分析基础模型能力的基线分析

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

Scale AI

Will LeVine, Benjamin Pikus, Anthony Chen, Sean Hendryx

AI总结:

本文评估奖励模型在分布偏移下的准确率与校准,发现其对响应偏移更敏感,并适配分类中的 OOD 检测技术识别提示与响应偏移。

摘要:

基础模型,尤其是大语言模型(LLM),近来获得了广泛关注与采用。基于人类反馈的强化学习(RLHF)涉及训练一个奖励模型以捕捉期望行为,随后用该模型对齐 LLM。这些奖励模型还在推理时被用于评估 LLM 响应对这些期望行为的遵循程度。然而,衡量这些奖励模型对分布偏移鲁棒性的工作很少。在本研究中,我们评估奖励模型的性能——通过准确率和校准(即准确率与置信度之间的一致性)来衡量——如何受到分布偏移的影响。我们展示了由 OOD 提示和响应导致的新型校准模式与准确率下降,并发现奖励模型对响应偏移比对提示偏移更敏感。此外,我们将一种常用于分类的 OOD 检测技术适配到奖励模型场景,以检测提示和响应中的这些分布偏移。

英文摘要:

Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑