arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.13709cs.CLcs.LG
收藏

理解直接偏好优化中的参考策略

Understanding Reference Policies in Direct Preference Optimization

Yale University(耶鲁大学);Shanghai Jiao Tong University(上海交通大学);Allen Institute for AI(艾伦人工智能研究所)

Yixin Liu, Pengfei Liu, Arman Cohan

AI总结:

本研究探讨了直接偏好优化(DPO)对参考策略的依赖性,发现DPO对KL散度约束强度敏感,且更强参考策略仅在相似时提升性能,为最佳实践提供了见解。

摘要:

直接偏好优化(DPO)已成为大语言模型(LLMs)指令微调的广泛使用的训练方法。在本研究中,我们探讨了DPO一个未被充分研究的方面——其对参考模型或参考策略的依赖性。此类参考策略通常实例化为待进一步微调的模型,它们非常重要,因为可能对DPO的有效性施加上限。因此,我们在本研究中探讨了三个相关的研究问题。首先,我们探索了DPO中KL散度约束的最佳强度,该约束惩罚偏离参考策略的行为,并发现DPO对该强度敏感。接着,我们通过提供DPO与相关学习目标之间的理论和实证比较,检验了DPO中来自参考策略的KL约束的必要性,证明了DPO在此受控设置中的优越性。此外,我们调查了DPO是否受益于更强的参考策略,发现更强的参考策略能提升性能,但仅当其与待微调模型相似时才成立。我们的发现突出了参考策略在DPO中的混淆作用,提供了最佳实践的见解,同时指出了未来研究的开放性问题。

英文摘要:

Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs). In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy. Such reference policies, typically instantiated as the model to be further fine-tuned, are important since they can impose an upper limit on DPO's effectiveness. Therefore, we address three related research questions in this work. First, we explore the optimal strength of the KL divergence constraint in DPO, which penalizes deviations from the reference policy, and find that DPO is sensitive to this strength. Next, we examine the necessity of the KL-constraint from the reference policies in DPO by providing both theoretical and empirical comparisons between DPO and related learning objectives, demonstrating DPO's superiority in this controlled setting. Additionally, we investigate whether DPO benefits from stronger reference policies, finding that a stronger reference policy can lead to improved performance, but only when it is similar to the model being fine-tuned. Our findings highlight the confounding role of reference policies in DPO and offer insights for best practices, while also identifying open research questions for future studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑