Reward models trained on human preference data have demonstrated strong effectiveness in aligning Large Language Models (LLMs) with human intent under the framework of Reinforcement Learning from Human Feedback (RLHF). However, RLHF remains vulnerable to reward hacking, where the policy exploits imperfections in the reward function rather than genuinely learning the intended behavior. Although significant efforts have been made to mitigate reward hacking, they predominantly focus on and evaluate in-distribution scenarios, where the training and testing data for the reward model share the same distribution. In this paper, we empirically show that state-of-the-art methods struggle in more challenging out-of-distribution (OOD) settings. We further demonstrate that incorporating fine-grained multi-attribute scores helps address this challenge. However, the limited availability of high-quality data often leads to weak performance of multi-objective reward functions, which can negatively impact overall performance and become the bottleneck. To address this issue, we propose a unified reward modeling framework that jointly trains Bradley--Terry (BT) single-objective and multi-objective regression-based reward functions using a shared embedding space. We theoretically establish a connection between the BT loss and the regression objective and highlight their complementary benefits. Specifically, the regression task enhances the single-objective reward function's ability to mitigate reward hacking in challenging OOD settings, while BT-based training improves the scoring capability of the multi-objective reward function, enabling a 7B model to outperform a 70B baseline. Extensive experimental results demonstrate that our framework significantly improves both the robustness and the scoring performance of reward models.
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与复杂系统决策智能实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司);Meituan Inc(美团公司);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Diverging Preferences: When do Annotators Disagree and do Models Know?
New York University(纽约大学);Allen Institute for Artificial Intelligence(人工智能研究院);NVIDIA(NVIDIA公司);University of Washington(华盛顿大学);University of Southern California(南加州大学)
On rankings in multiplayer games with an application to the game of Whist
Department of Mathematics, Naxys, Unamur, Belgium(比利时那慕尔大学纳西斯研究所数学系);Department of Physics and Astronomy, CSDC, University of Florence, Italy(意大利佛罗伦萨大学物理与天文学系CSDC)
Variance-aware Reward Modeling with Anchor Guidance
Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系);School of Statistics and Data Science, Shanghai University of Finance and Economics(上海财经大学统计与数据科学学院)
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
Department of Statistics, LSE(统计系,伦敦经济学院);Department of Mathematics, Tsinghua University(数学系,清华大学);School of Mathematics, University of Birmingham(数学学院,伯明翰大学);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)