Preference learning (PL) with large language models (LLMs) aims to align the LLMs' generations with human preferences. Previous work on reinforcement learning from human feedback (RLHF) has demonstrated promising results in in-distribution PL. However, due to the difficulty of obtaining human feedback, discretely training reward models for every encountered distribution is challenging. Thus, out-of-distribution (OOD) PL is practically useful for enhancing the generalization ability of LLMs with limited preference feedback. This work addresses OOD PL by optimizing a general reward model through a meta-learning approach. During meta-training, a bilevel optimization algorithm is utilized to learn a reward model capable of guiding policy learning to align with human preferences across various distributions. When encountering a test distribution, the meta-test procedure conducts regularized policy optimization using the learned reward model for PL. We theoretically demonstrate the convergence rate of the bilevel optimization algorithm under reasonable assumptions. Additionally, we conduct experiments on two text generation tasks across 20 held-out domains and outperform a variety of strong baselines across various evaluation metrics.
RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Jiao Tong University（上海交通大学）；Carnegie Mellon University（卡内基梅隆大学）；Jilin University（吉林大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
CREAM: Consistency Regularized Self-Rewarding Language Models
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Nanyang Technological University（南洋理工大学）；National University of Singapore（国立新加坡大学）；Microsoft Research（微软研究院）
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程保罗·G·艾伦学校）；Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根马克斯·普朗克智能系统研究所）；Jinesis Lab, University of Toronto & Vector Institute（多伦多大学Jinesis实验室及向量研究所）
Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization
Saarland University（萨尔兰大学）；Independent Researcher（独立研究者）；Bielefeld University（比勒菲尔德大学）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Meituan Inc.（美团公司）；NiuTrans Research（牛译研所）；CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS（中国科学院行为科学重点实验室）；Kunming University of Science and Technology（昆明理工大学）
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China（清华大学信息科学技术学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Department of Computer Science, University of California, Los Angeles, California, USA（计算机科学系，加州大学洛杉矶分校，美国，加州）
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Department of Computer Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机科学与数据科学学院）
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（国立新加坡大学）；The University of Tokyo（东京大学）；MiroMind（MiroMind公司）；Institute for Infocomm Research, A*Star, Singapore（新加坡*星研究院信息与通信研究所）
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
The Graduate University for Advanced Studies (SOKENDAI)（高级研究大学（SOKENDAI））；National Institute of Informatics（国家信息研究所）；The Asahi Shimbun Company（朝日新闻公司）；The University of Tokyo（东京大学）
