arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.07375cs.LGcs.CL
收藏

Bradley-Terry与多目标奖励建模是互补的

Bradley-Terry and Multi-Objective Reward Modeling Are Complementary

The Pennsylvania State University(宾夕法尼亚州立大学);Amazon(亚马逊);Havard University(哈佛大学)

Zhiwei Zhang, Hui Liu, Xiaomin Li, Zhenwei Dai, Jingying Zeng, Fali Wang, Minhua Lin, Ramraj Chandradevan, Zhen Li, Chen Luo, Xianfeng Tang, Qi He, Suhang Wang

AI总结:

针对RLHF在分布外易受奖励黑客攻击的问题,提出联合训练BT单目标与多目标回归奖励的统一框架,显著提升了奖励模型在OOD设置下的鲁棒性和评分能力。

摘要:

基于人类偏好数据训练的奖励模型在人类反馈强化学习(RLHF)框架下,已证明在使大型语言模型(LLMs)与人类意图对齐方面具有强大效果。然而,RLHF仍然容易受到奖励黑客攻击,即策略利用奖励函数中的缺陷,而非真正学习预期行为。尽管已付出大量努力来缓解奖励黑客攻击,但它们主要集中于评估分布内场景,即奖励模型的训练和测试数据共享相同分布。在本文中,我们经验性地证明了最先进的方法在更具挑战性的分布外(OOD)设置中表现挣扎。我们进一步证明,结合细粒度的多属性分数有助于应对这一挑战。然而,高质量数据的有限可用性通常导致多目标奖励函数性能较弱,这可能对整体性能产生负面影响并成为瓶颈。为解决此问题,我们提出了一个统一的奖励建模框架,使用共享嵌入空间联合训练Bradley-Terry(BT)单目标和多目标基于回归的奖励函数。我们在理论上建立了BT损失与回归目标之间的联系,并突出了它们互补的益处。具体而言,回归任务增强了单目标奖励函数在具有挑战性的OOD设置中缓解奖励黑客攻击的能力,而基于BT的训练提高了多目标奖励函数的评分能力,使7B模型能够超越70B基线。广泛的实验结果表明,我们的框架显著提升了奖励模型的鲁棒性和评分性能。

英文摘要:

Reward models trained on human preference data have demonstrated strong effectiveness in aligning Large Language Models (LLMs) with human intent under the framework of Reinforcement Learning from Human Feedback (RLHF). However, RLHF remains vulnerable to reward hacking, where the policy exploits imperfections in the reward function rather than genuinely learning the intended behavior. Although significant efforts have been made to mitigate reward hacking, they predominantly focus on and evaluate in-distribution scenarios, where the training and testing data for the reward model share the same distribution. In this paper, we empirically show that state-of-the-art methods struggle in more challenging out-of-distribution (OOD) settings. We further demonstrate that incorporating fine-grained multi-attribute scores helps address this challenge. However, the limited availability of high-quality data often leads to weak performance of multi-objective reward functions, which can negatively impact overall performance and become the bottleneck. To address this issue, we propose a unified reward modeling framework that jointly trains Bradley--Terry (BT) single-objective and multi-objective regression-based reward functions using a shared embedding space. We theoretically establish a connection between the BT loss and the regression objective and highlight their complementary benefits. Specifically, the regression task enhances the single-objective reward function's ability to mitigate reward hacking in challenging OOD settings, while BT-based training improves the scoring capability of the multi-objective reward function, enabling a 7B model to outperform a 70B baseline. Extensive experimental results demonstrate that our framework significantly improves both the robustness and the scoring performance of reward models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑