arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.07599cs.CL
收藏

DPO-Shift:平移直接偏好优化的分布

DPO-Shift: Shifting the Distribution of Direct Preference Optimization

David S. Hippocampus Department of Computer Science(计算机科学系)

Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, Xiao Li

AI总结:

本文提出 DPO-Shift 方法,通过可控平移被选择概率分布来缓解 DPO 训练中的似然位移问题,并在理论与实验上验证了其在提高被选择概率与牺牲奖励边际之间的权衡及下游任务优势。

摘要:

直接偏好优化(DPO)及其变体在使语言模型与人类偏好对齐方面日益流行。这些方法旨在教会模型更好地区分被选择(或偏好)的响应与被拒绝(或不偏好)的响应。然而,先前的研究已经发现,在训练过程中被选择响应的概率往往会下降,这一现象被称为似然位移。为应对这一挑战,本文引入了 DPO-Shift,以可控地平移被选择概率的分布。随后,我们证明 DPO-Shift 在提高被选择概率与牺牲奖励边际之间存在一种根本性的权衡,这一结论得到了理论分析和实验验证的支持。此外,我们在 MT-Bench 等下游任务以及一个设计的胜率实验中证明了 DPO-Shift 相对于 DPO 的优越性。我们相信,本研究表明 DPO 的似然位移问题可以通过一个简单且有理论依据的解决方案得到有效缓解。我们的代码可在 https://github.com/Meaquadddd/DPO-Shift 获取。

英文摘要:

Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce DPO-Shift to controllably shift the distribution of the chosen probability. Then, we show that DPO-Shift exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of DPO-Shift over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑