将大语言模型(LLMs)与人类偏好对齐已受到广泛关注,其中近端策略优化(PPO)是标准但计算成本高昂的方法,而直接偏好优化(DPO)则是更高效的替代方案。尽管DPO实现简单,但它在最先进的大语言模型中仍未得到充分利用,这表明其可能存在局限性。在本工作中,我们重新审视DPO,分析其理论基础和实证表现以弥合这一差距。我们识别出DPO学习过程中出现的三个关键性质,统称为3D性质:被拒绝响应似然的急剧下降(Drastic drop)、退化为响应抑制(Degradation into response suppression)以及对未见响应的分散效应(Dispersion effect)。我们证明这些问题源于DPO的优化动力学,其中被选择响应和被拒绝响应的梯度之间的相互作用导致不稳定性。我们的发现得到了受控玩具模型和真实世界大语言模型任务实验的支持,包括数学问题求解和指令遵循。为应对这些挑战,我们提出了简单的正则化技术,以提高训练稳定性和性能。此外,我们考察了偏好数据分布如何影响DPO的有效性,为对齐模型如何处理域外(OOD)数据提供了见解。我们的工作将这些观察结果与更广泛的研究联系起来,并为DPO的局限性提供了理论解释。我们希望这些见解能够指导无奖励模型偏好学习的未来发展,使其更接近基于奖励模型的方法。
英文摘要:
Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties, termed 3D properties, that emerge from DPO's learning process: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. We show that these issues arise from DPO's optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO's effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO's limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions
South China University of Technology(华南理工大学);Tencent Financial Technology(腾讯金融科技);The Hong Kong University of Science and Technology(香港科学与技术大学);Soochow University(苏州大学);Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University(浙江省智能教育技术与应用重点实验室,浙江师范大学)
CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
National Engineering Research Center for Big Data Technology and Systems(大数据技术与系统国家工程研究中心);Services Computing Technology and System Lab(服务计算技术与系统实验室);Cluster and Grid Computing Lab(集群与网格计算实验室);School of Computer Science and Technology(计算机科学与技术学院);Huazhong University of Science and Technology(华中科技大学);Wuhuan University(吴奂大学);Zhejiang University(浙江大学)
Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities
Guelma University(盖勒马大学);Technology Innovation Institute(技术创新研究院);Concordia University(康考迪亚大学);University of Oslo(奥斯陆大学);Khalifa University of Science and Technology(哈利法科技大学)
SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
Shanghai Science and Intelligence Institute, Shanghai, China(上海科学与智能研究所);Fudan University, Shanghai, China(复旦大学);Australian Institute for Machine Learning, The University of Adelaide(澳大利亚机器学习研究所,阿德莱德大学)