从RLHF到直接对齐：大型语言模型偏好学习的理论统一
From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
AI总结：
本文提出了一种理论统一框架，将大型语言模型的偏好学习方法归结为三个正交轴上的原则性选择，揭示了不同方法之间的本质联系。
摘要：
将大型语言模型（LLMs）与人类偏好对齐已成为安全和有益的人工智能部署的关键。尽管强化学习从人类反馈（RLHF）建立了主导范式，但替代方法的激增——直接偏好优化（DPO）、身份偏好优化（IPO）、卡尼曼-特维斯基优化（KTO）、简单偏好优化（SimPO）等——使从业者在方法选择上缺乏明确指导。本文提供了一种偏好学习方法的理论统一，揭示出看似多样化的技术实际上可以归结为三个正交轴上的原则性选择：（I）偏好模型（目标背后的概率模型）、（II）正则化机制（如何控制偏离参考策略）以及（III）数据分布（在线 vs. 离线学习和覆盖要求）。我们为每个轴制定了精确的定义和定理，建立了关键结果，包括在线与离线方法的覆盖分离、奖励过优化的缩放定律，以及直接对齐方法失败的条件。我们的分析表明，失败模式——长度黑客、模式崩溃、似然位移——源于特定且可预测的设计选择组合。我们综合了50多篇论文的实证发现，并提供了从业者的方法选择决策指南。该框架将偏好学习从经验艺术转变为理论基础的学科。
英文摘要：
Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a \textit{theoretical unification} of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: \textbf{(I) Preference Model} (what likelihood model underlies the objective), \textbf{(II) Regularization Mechanism} (how deviation from reference policies is controlled), and \textbf{(III) Data Distribution} (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
