发表机构
Northeastern University（东北大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对DPO在偏好边际增大时损失敏感度下降的问题，提出LSC-DPO动态调节学习信号，并通过信号预算补偿减少初始化差异，实验优于现有基线。
AI 中文摘要
直接偏好优化（DPO）已成为一种标准的无需奖励模型的范式，用于将语言模型与偏好数据对齐。然而，随着训练过程中缩放偏好边际的增长，逻辑DPO损失对进一步变化的敏感度逐渐降低。我们从损失层面的几何视角研究DPO，并将sigmoid因子识别为一种学习信号，它刻画了目标的局部敏感性。基于这一观点，我们提出了学习信号控制的直接偏好优化（LSC-DPO），它动态地调节目标区域附近的学习信号。对数空间分析为稳定跟踪目标学习信号区域建立了条件。在AlpacaEval 2、MT-Bench和Anthropic-HH上的实验表明，LSC-DPO始终优于DPO和强偏好优化基线。我们进一步发现，不同的系数初始化会引发不同的瞬态学习信号轨迹，即使它们后续的信号水平变得相似。基于这一观察，我们推导出一种信号预算补偿规则，该规则调整目标学习信号以补偿这些瞬态差异。由此产生的补偿显著降低了不同系数初始化之间的性能差异。
英文摘要
Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.
Comments27 pages, 15 figures, 14 tables
