arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00603cs.LG
收藏

直接偏好优化与评分信息:实用算法和可证明的收益

Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains

EPFL(苏黎世联邦理工学院);Amazon AGI(亚马逊人工智能实验室);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Cornell University(康奈尔大学);Qualcomm AI Research(高通人工智能研究)

Luca Viano, Ruida Zhou, Yifan Sun, Mahdi Namazifar, Volkan Cevher, Shoham Sabach, Mohammad Ghavamzadeh

AI总结:

本文提出利用评分间隙信息改进直接偏好优化算法,通过理论证明和实验验证,在准确评分间隙条件下实现更快的统计速率,并在多种LLM和基准上表现优异。

摘要:

直接偏好优化(DPO)算法类作为一种解决基础模型对齐问题的有希望的方法已经出现。这些算法以成对偏好作为有限反馈形式来工作,并通过微调模型来与这些偏好对齐,而不需要显式学习奖励模型。尽管这些算法所使用的反馈形式使数据收集过程变得容易且相对更准确,但其在响应质量方面的模糊性可能有负面影响。例如,在执行这些算法时,首选(不首选)响应的似然度的减少(增加)是否可以被解释为积极或消极现象并不明确。在本文中,我们研究如何设计能够利用评分间隙额外信息的算法,该信息告知学习者所选响应比被拒绝的响应更好多少。我们提出了新的算法,这些算法在存在准确评分间隙信息的情况下可以实现比DPO更快的统计速率。此外,我们理论证明并实证显示,我们的算法在评分间隙不准确的情况下表现稳健。最后,我们展示了我们的方法在广泛的LLM和评估基准上与多种DPO风格算法相比的优异表现。

英文摘要:

The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with very limited feedback in the form of pairwise preferences and fine-tune models to align with these preferences without explicitly learning a reward model. While the form of feedback used by these algorithms makes the data collection process easy and relatively more accurate, its ambiguity in terms of the quality of responses could have negative implications. For example, it is not clear if a decrease (increase) in the likelihood of preferred (dispreferred) responses during the execution of these algorithms could be interpreted as a positive or negative phenomenon. In this paper, we study how to design algorithms that can leverage additional information in the form of rating gap, which informs the learner how much the chosen response is better than the rejected one. We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information. Moreover, we theoretically prove and empirically show that the performance of our algorithms is robust to inaccuracy in rating gaps. Finally, we demonstrate the solid performance of our methods in comparison to a number of DPO-style algorithms across a wide range of LLMs and evaluation benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑