迭代长度正则化直接偏好优化：将7B语言模型提升至GPT-4水平的案例研究
Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level
AI总结：
针对迭代DPO提升回复质量却增加冗长性的问题，本文提出iLR-DPO通过长度正则惩罚约束回复长度，使7B模型在AlpacaEval 2.0等基准上达到GPT-4级表现。
摘要：
直接偏好优化（DPO）是使语言模型与人类偏好对齐的标准方法，传统上应用于离线偏好。近期研究表明，DPO可受益于使用经过训练的奖励模型标注的在线偏好进行迭代训练。在本工作中，我们发现原始迭代DPO的一个缺陷——回复质量的提升可能导致冗长性增加。为解决这一问题，我们提出迭代长度正则化DPO（iLR-DPO），以惩罚回复长度。实验结果表明，iLR-DPO能够增强7B模型，使其在不增加冗长性的情况下达到与GPT-4相当的性能。具体而言，我们的7B模型在AlpacaEval 2.0上针对GPT-4 Preview取得50.5%的长度受控胜率，并在包括MT-Bench、Arena-Hard和OpenLLM Leaderboard在内的标准基准测试中表现优异。这些结果证明了迭代DPO在使语言模型与人类反馈对齐方面的有效性。
英文摘要：
Direct Preference Optimization (DPO), a standard method for aligning language models with human preferences, is traditionally applied to offline preferences. Recent studies show that DPO benefits from iterative training with online preferences labeled by a trained reward model. In this work, we identify a pitfall of vanilla iterative DPO - improved response quality can lead to increased verbosity. To address this, we introduce iterative length-regularized DPO (iLR-DPO) to penalize response length. Our empirical results show that iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing verbosity. Specifically, our 7B model achieves a $50.5\%$ length-controlled win rate against $\texttt{GPT-4 Preview}$ on AlpacaEval 2.0, and excels across standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard. These results demonstrate the effectiveness of iterative DPO in aligning language models with human feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
