稳定偏好优化：一种针对灾难性偏好转移的双层方法
Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
AI总结：
本文提出稳定偏好优化框架，通过约束偏好学习在安全对齐区域，解决灾难性偏好转移问题，提升偏好学习方法的稳定性和性能。
摘要：
直接偏好学习已成为偏好优化的主导离线范式。大多数这些方法基于布拉德利-蒂尔模型进行成对偏好排名，该模型直接将语言模型与人类偏好对齐。先前的工作观察到一种反直觉现象，称为似然位移，即在训练过程中，受青睐响应的绝对概率同时下降。我们证明这种位移可能导致更严重的失败模式，即定义为灾难性偏好转移，其中失去的偏好概率质量意外地转移到分布外（OOD）响应。这种失败模式是所有BT式直接偏好学习方法的关键限制，由于在约束性判别对齐与生成基础能力之间存在根本冲突，最终导致严重性能下降（例如，SimPO在推理准确性上从73.5%下降到37.5%）。我们从概率演变的角度分析现有BT式方法，并理论证明这些方法过度依赖模型初始化，可能导致偏好转移。为了解决这些反直觉行为，我们提出了一个理论支持的稳定偏好优化（SPO）框架，该框架将偏好学习限制在安全对齐区域内。实证评估表明，SPO有效稳定并提升了现有BT式偏好学习方法的性能。SPO为偏好学习目标的设计提供了新的见解，并为更可靠和可解释的语言模型对齐开辟了新途径。
英文摘要：
Direct Preference Learning has emerged as a dominant offline paradigm for preference optimization. Most of these methods are based on the Bradley-Terry (BT) model for pairwise preference ranking, which directly aligns language model with human preference. Prior work has observed a counter-intuitive phenomenon termed likelihood displacement, where the absolute probability of preferred responses decreases simultaneously during training. We demonstrate that such displacement can lead to a more devastating failure mode, which we defined as \textit{Catastrophic Preference Shift}, where the lost preference probability mass inadvertently shifts toward out-of-distribution (OOD) responses. Such a failure mode is a key limitation shared across BT-style direct preference learning methods, due to the fundamental conflict between the unconstrained discriminative alignment and generative foundational capabilities, ultimately leading to severe performance degradation (e.g., SimPO suffers a significant drop in reasoning accuracy from 73.5\% to 37.5\%). We analyze existing BT-style methods from the probability evolution perspective and theoretically prove that these methods exhibit over-reliance on model initialization and can lead to preference shift. To resolve these counter-intuitive behaviors, we propose a theoretically grounded Stable Preference Optimization (SPO) framework that constrains preference learning within a safe alignment region. Empirical evaluations demonstrate that SPO effectively stabilizes and enhances the performance of existing BT-style preference learning methods. SPO provides new insights into the design of preference learning objectives and opens up new avenues towards more reliable and interpretable language model alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
