arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.13787cs.LG
收藏

RewardBench:评估语言建模中的奖励模型

RewardBench: Evaluating Reward Models for Language Modeling

Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Harvard Law(哈佛法学院)

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

AI总结:

本文提出RewardBench基准数据集与代码库,通过涵盖聊天、推理和安全的提示-选中-拒绝三元组评估多种奖励模型,揭示其在拒绝倾向、推理和指令遵循方面的不足,以深化对RLHF过程的理解。

摘要:

奖励模型(RMs)是成功利用RLHF将预训练模型与人类偏好对齐的关键所在,然而相对而言,专注于评估这些模型的研究却很少。评估奖励模型提供了一个机会,可以理解用于语言模型对齐的不透明技术以及其中嵌入的价值观。在围绕奖励模型的新兴开源社区中,用于奖励模型训练和理解的资源十分稀缺。为了增进对奖励模型的科学理解,我们提出了RewardBench,一个用于评估的基准数据集和代码库。RewardBench数据集是涵盖聊天、推理和安全领域的提示-选中-拒绝三元组集合,用于基准测试奖励模型在具有挑战性、结构化和分布外查询上的表现。我们为奖励模型创建了特定的比较数据集,这些数据集具有微妙但可验证的理由(例如错误、不正确的事实),说明为什么一个答案应优先于另一个答案。在RewardBench排行榜上,我们评估了使用多种方法训练的奖励模型,例如分类器的直接MLE训练和直接偏好优化(DPO)的隐式奖励建模。我们提出了许多关于各种奖励模型的拒绝倾向、推理局限性和指令遵循缺陷的发现,以更好地理解RLHF过程。

英文摘要:

Reward models (RMs) are at the crux of successfully using RLHF to align pretrained models to human preferences, yet there has been relatively little study that focuses on evaluation of those models. Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. Resources for reward model training and understanding are sparse in the nascent open-source community around them. To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation. The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. We create specific comparison datasets for RMs that have subtle, but verifiable reasons (e.g. bugs, incorrect facts) why one answer should be preferred to another. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO). We present many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑