Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.
RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);Jilin University(吉林大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
CREAM: Consistency Regularized Self-Rewarding Language Models
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Microsoft Research(微软研究院)
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学计算机科学与工程保罗·G·艾伦学校);Max Planck Institute for Intelligent Systems, Tübingen, Germany(德国图宾根马克斯·普朗克智能系统研究所);Jinesis Lab, University of Toronto & Vector Institute(多伦多大学Jinesis实验室及向量研究所)
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University(萨尔兰大学);Independent Researcher(独立研究者);Bielefeld University(比勒菲尔德大学);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息研究所)
The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems
Berkeley AI Safety Initiative (BASIS) UC Berkeley(伯克利人工智能安全计划(BASIS)伯克利大学);Department of Electrical and Computer Engineering Johns Hopkins University(电气与计算机工程系约翰霍普金斯大学)
Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Meituan Inc.(美团公司);NiuTrans Research(牛译研所);CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS(中国科学院行为科学重点实验室);Kunming University of Science and Technology(昆明理工大学)
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Department of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院)
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
Singapore University of Technology and Design(新加坡科技设计大学);National University of Singapore(国立新加坡大学);The University of Tokyo(东京大学);MiroMind(MiroMind公司);Institute for Infocomm Research, A*Star, Singapore(新加坡*星研究院信息与通信研究所)
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
The Graduate University for Advanced Studies (SOKENDAI)(高级研究大学(SOKENDAI));National Institute of Informatics(国家信息研究所);The Asahi Shimbun Company(朝日新闻公司);The University of Tokyo(东京大学)