带偏移量的直接偏好优化
Direct Preference Optimization with an Offset
AI总结：
本文提出带偏移量的直接偏好优化（ODPO），通过要求偏好响应与不偏好响应的似然差超过基于偏好强度确定的偏移值，解决了DPO平等对待所有偏好对的问题，并在偏好对有限时显著提升语言模型对齐效果。
摘要：
直接偏好优化（DPO）是一种成功的微调策略，用于使大型语言模型与人类偏好对齐，而无需训练奖励模型或采用强化学习。最初提出的DPO依赖于二元偏好数据，并微调语言模型以提高被偏好响应相对于不被偏好响应的似然。然而，并非所有偏好对都是等同的。有时，被偏好的响应仅比不被偏好的响应略好。在其他情况下，偏好则强得多。例如，如果一个响应包含有害或有毒内容，标注者会对该响应产生强烈偏好。在本文中，我们提出了DPO的一种推广形式，称为带偏移量的DPO（ODPO），它在微调期间并不平等地对待每一个偏好对。直观地说，ODPO要求被偏好响应与不被偏好响应的似然之差大于一个偏移值。该偏移量根据一个响应相对于另一个响应被偏好的程度来确定。我们在各种任务上的实验表明，ODPO在对齐语言模型方面显著优于DPO，尤其是在偏好对数量有限的情况下。
英文摘要：
Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
