arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07319cs.LGcs.AIcs.CL
收藏

ODIN:解耦的奖励缓解 RLHF 中的奖励作弊

ODIN: Disentangled Reward Mitigates Hacking in RLHF

University of YYY(YYY大学);Institute of WWW(WWW研究所)

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro

AI总结:

研究RLHF中基于响应长度的奖励作弊问题,提出通过联合训练与长度相关及去相关的两个线性头并丢弃长度头来改进奖励模型,有效消除奖励与长度的相关性并显著提升策略质量。

摘要:

在这项工作中,我们研究了关于响应长度的奖励作弊问题,这是大型语言模型(LLMs)上基于人类反馈的强化学习(RLHF)中出现的一个挑战。来自 LLMs 的一个格式良好、冗长但帮助较小的响应往往能欺骗 LLMs 甚至人类评估者,从而获得高分。同样的问题也适用于 RL 中的一些奖励模型。为了解决训练和评估中的挑战,我们建立了一个更可靠的评估协议,用于比较不同的训练配置,该协议检查通过改变训练超参数获得的 LLM 评估分数和响应长度之间的权衡。基于此评估,我们进行了大规模研究,其结果为 RL 中使用的超参数和技巧在缓解长度偏差方面的有效性提供了见解。我们进一步提出通过在共享特征表示上联合训练两个线性头来改进奖励模型以预测奖励,其中一个被训练为与长度相关,另一个被训练为与长度去相关,从而更关注实际内容。然后,我们在 RL 中丢弃长度头以防止对长度的奖励作弊。实验表明,我们的方法几乎消除了奖励与长度的相关性,并在显著程度上改进了所获得的策略。

英文摘要:

In this work, we study the issue of reward hacking on the response length, a challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on LLMs. A well-formatted, verbose but less helpful response from the LLMs can often deceive LLMs or even human evaluators to achieve high scores. The same issue also holds for some reward models in RL. To address the challenges in both training and evaluation, we establish a more reliable evaluation protocol for comparing different training configurations, which inspects the trade-off between LLM evaluation score and response length obtained by varying training hyperparameters. Based on this evaluation, we conduct large-scale studies, where the results shed insights into the efficacy of hyperparameters and tricks used in RL on mitigating length bias. We further propose to improve the reward model by jointly training two linear heads on shared feature representations to predict the rewards, one trained to correlate with length, and the other trained to decorrelate with length and therefore focus more on the actual content. We then discard the length head in RL to prevent reward hacking on length. Experiments demonstrate that our approach almost eliminates the reward correlation with length, and improves the obtained policy by a significant margin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑