RLHF中的准确率悖论：当更好的奖励模型不能产生更好的语言模型
The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models
AI总结：
本文通过QA-FEEDBACK数据集和Longformer奖励模型实验，揭示RLHF中的准确率悖论：使用中等准确率奖励模型训练的语言模型优于高准确率奖励模型引导的模型，挑战了更强奖励模型必产生更好语言模型的观念。
摘要：
人类反馈强化学习（RLHF）通过使语言模型与人类期望对齐，显著提升了自然语言处理（NLP）的效果。这种对齐过程中的一个关键因素是训练期间使用的奖励模型的强度。本研究探讨了更强的奖励模型是否总能产生更好的语言模型。在本文中，通过使用QA-FEEDBACK数据集和基于Longformer的奖励模型，在相关性、事实性和完整性任务上进行实验，我们揭示了一个令人惊讶的悖论：使用中等准确率的奖励模型训练的语言模型，优于由高准确率奖励模型引导的语言模型。这挑战了“更强的奖励模型总是能产生更好的语言模型”这一普遍观念，并为未来研究驱动模型性能的关键因素以及如何选择最合适的奖励模型开辟了新途径。代码及更多细节见https://github.com/EIT-NLP/AccuracyParadox-RLHF。
英文摘要：
Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations. A critical factor in this alignment is the strength of reward models used during training. This study explores whether stronger reward models invariably lead to better language models. In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones. This challenges the widely held belief that stronger reward models always lead to better language models, and opens up new avenues for future research into the key factors driving model performance and how to choose the most suitable reward models. Code and additional details are available at https://github.com/EIT-NLP/AccuracyParadox-RLHF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
