Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training.
In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.
UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心);Department of Computer Science(计算机科学系);University College London(伦敦大学学院);UK AI Security Institute(英国人工智能安全研究所);Department of Computer Science and Engineering(计算机科学与工程系);University of Bologna(博洛尼亚大学)
Reward Difference Optimization For Sample Reweighting In Offline RLHF
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);The Chinese University of Hong Kong(香港中文大学);New Jersey Institute of Technology(新泽西理工学院)
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Kuaishou Technology(快手科技);University of Chinese Academy of Sciences(中国科学院大学)
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Zhejiang University(浙江大学);Xiaohongshu Inc(小红书公司);National University of Singapore(新加坡国立大学);New York University(纽约大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);The State Key Lab of Brain-Machine Intelligence, Zhejiang University(浙江大学脑机智能国家重点实验室);LLM Safety Centre, Beijing Academy of Artificial Intelligence(北京人工智能研究院大语言模型安全中心);Center for AI Safety and Governance, Peking University(北京大学人工智能安全与治理中心)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong(香港中文大学);Microsoft Research Asia(微软亚洲研究院);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration
Department of Electrical & Computer Engineering(电气与计算机工程系);Boston University(波士顿大学);Department of Computer Science(计算机科学系);Cornell University(康奈尔大学);Faculty of Computing & Data Sciences(计算与数据科学学院)
S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computer Science and Artificial Intelligence, Zhengzhou University(郑州大学计算机科学与人工智能学院)