Since the debut of DPO, it has been shown that aligning a target LLM with human preferences via the KL-constrained RLHF loss is mathematically equivalent to a special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has a 1:1 linear relationship with the true reward. However, we identify a significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from a well-known issue of the underlying Bradley-Terry preference model: it does not always have a unique maximum likelihood estimator (MLE). Consequently,the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we propose an energy-based model (EBM) that always has a unique MLE, inherently satisfying the linearity requirement. To approximate the MLE in practice, we propose a contrastive loss named Energy Preference Alignment (EPA), wherein each positive sample is contrasted against one or more strong negatives as well as many free weak negatives. Theoretical properties of our EBM enable the approximation error of EPA to almost surely vanish when a sufficient number of negatives are used. Empirically, we demonstrate that EPA consistently delivers better performance on open benchmarks compared to DPO, thereby showing the superiority of our EBM.
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
Diverging Preferences: When do Annotators Disagree and do Models Know?
New York University(纽约大学);Allen Institute for Artificial Intelligence(人工智能研究院);NVIDIA(NVIDIA公司);University of Washington(华盛顿大学);University of Southern California(南加州大学)
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与复杂系统决策智能实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司);Meituan Inc(美团公司);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
On rankings in multiplayer games with an application to the game of Whist
Department of Mathematics, Naxys, Unamur, Belgium(比利时那慕尔大学纳西斯研究所数学系);Department of Physics and Astronomy, CSDC, University of Florence, Italy(意大利佛罗伦萨大学物理与天文学系CSDC)
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
University of California, Berkeley(加州大学伯克利分校);Center for Advanced Intelligence Project, RIKEN(RIKEN先进智能项目中心);The Institute of Statistical Mathematics and the Graduate University for Advanced Studies(统计数理研究所与综合研究大学院大学);Tohoku University(东北大学)
Variance-aware Reward Modeling with Anchor Guidance
Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系);School of Statistics and Data Science, Shanghai University of Finance and Economics(上海财经大学统计与数据科学学院)
Department of Statistics(统计系);LSE London, UK(伦敦大学学院);Department of Mathematics(数学系);Tsinghua University(清华大学);School of Design LCC, UAL London, UK(伦敦艺术大学设计学院);Department of Engineering Science(工程科学系);University of Oxford(牛津大学)
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
Department of Statistics, LSE(统计系,伦敦经济学院);Department of Mathematics, Tsinghua University(数学系,清华大学);School of Mathematics, University of Birmingham(数学学院,伯明翰大学);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)