直接偏好优化的新要求
New Desiderata for Direct Preference Optimization
AI总结：
针对现有直接偏好优化（DPO）方法在插值能力、响应正则化及约束处理上的缺陷，该研究提出新评估标准，并设计了可证明缓解这些局限的类DPO损失函数，且通过实验验证了分析结论。
摘要：
过去的大语言模型通常依赖某种形式的人类反馈强化学习（RLHF），以使模型响应更好地契合人类偏好。然而，由于在实现这些RLHF流程时经常观察到不稳定性，近期人们引入了多种重参数化技术，以省去单独学习RL奖励模型的需求。取而代之的是，通过最小化单个闭式训练目标来实现针对人类偏好的直接微调，这一过程最初被称为直接偏好优化（DPO），随后又出现了若干值得关注的衍生方法。尽管现有DPO方法在某些实际场景中有效，我们提出了新的评估标准，用以凸显其尚未解决的缺陷：现有DPO方法在预训练参考模型与人类偏好的实证度量之间进行插值的能力存在不足，且在低质量与高质量响应的正则化方式、约束处理方面存在不可避免的权衡。基于这些见解，我们提出了一种类DPO的替代损失函数，可证明能缓解这些局限性。实证结果证实了我们分析中的关键结论。
英文摘要：
Large language models in the past have typically relied on some form of reinforcement learning with human feedback (RLHF) to better align model responses with human preferences. However, because of oft-observed instabilities when implementing these RLHF pipelines, various reparameterization techniques have recently been introduced to sidestep the need for separately learning an RL reward model. Instead, directly fine-tuning for human preferences is achieved via the minimization of a single closed-form training objective, a process originally referred to as direct preference optimization (DPO) and followed by several notable descendants. Although effective in certain real-world settings, we introduce new evaluation criteria that serve to highlight unresolved shortcomings in the ability of existing DPO methods to interpolate between a pre-trained reference model and empirical measures of human preferences, as well as unavoidable trade-offs in how low- and high-quality responses are regularized and constraints are handled. Our insights then motivate an alternative DPO-like loss that provably mitigates these limitations. Empirical results serve to corroborate notable aspects of our analyses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
