基于人类反馈的强化学习（Reinforcement Learning from Human Feedback, RLHF）已成为实现大语言模型（large language models, LLMs）与人类偏好对齐的核心工具。直接偏好优化（Direct Preference Optimization, DPO）作为最主流的方法之一，将RLHF建模为策略优化问题，无需显式估计奖励函数。它解决了两步法的稳定性与效率问题，这类方法通常先估计奖励函数，再通过近端策略优化（proximal policy optimization, PPO）优化策略。由于RLHF本质上是一个优化问题，且动量技术在理论和实证上均能加速优化，一个自然的问题随之产生：动量能否加速RLHF？本文给出了肯定的答案。具体而言，我们首先证明迭代偏好优化方法可被视为一种近端点方法。基于这一观察，我们提出了通用的加速偏好优化（Accelerated Preference Optimization, APO）框架，该框架统一了多种现有偏好优化算法，并采用Nesterov动量技术加快LLM的对齐速度。理论上，我们证明APO相比标准迭代偏好优化方法（包括DPO和自博弈偏好优化（Self-Play Preference Optimization, SPPO））能实现更快的收敛速度。实验上，我们在AlpacaEval 2.0基准上验证了APO在RLHF任务中优于DPO、迭代DPO及其他强基线方法。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO), one of the most popular approaches, formulates RLHF as a policy optimization problem without explicitly estimating the reward function. It overcomes the stability and efficiency issues of two-step approaches, which typically involve first estimating the reward function and then optimizing the policy via proximal policy optimization (PPO). Since RLHF is essentially an optimization problem, and it is well-known that momentum techniques can accelerate optimization both theoretically and empirically, a natural question arises: Can RLHF be accelerated by momentum? This paper answers this question in the affirmative. In detail, we first show that the iterative preference optimization method can be viewed as a proximal point method. Based on this observation, we propose a general Accelerated Preference Optimization (APO) framework, which unifies many existing preference optimization algorithms and employs Nesterov's momentum technique to speed up the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a faster convergence rate than the standard iterative preference optimization methods, including DPO and Self-Play Preference Optimization (SPPO). Empirically, we show the superiority of APO over DPO, iterative DPO, and other strong baselines for RLHF on the AlpacaEval 2.0 benchmark.
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering（计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications（新一代人工智能技术及其交叉应用关键实验室）；Ministry of Education（教育部）
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China（清华大学信息科学技术学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Department of Computer Science, University of California, Los Angeles, California, USA（计算机科学系，加州大学洛杉矶分校，美国，加州）
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学计算机科学系）；Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学电气与计算机工程系）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；Appier AI Research（Appier人工智能研究院）；Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）
FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Communication Content Cognition（通信内容认知国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Tencent Inc.（腾讯公司）
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
Harbin Institute of Technology（哈尔滨工业大学）；Beijing Normal University - Hong Kong Baptist University United International College（北京师范大学-香港浸会大学联合国际学院）；Qilu University of Technology（齐鲁工业大学）
