直接偏好优化中的长度脱敏
Length Desensitization in Direct Preference Optimization
AI总结：
针对直接偏好优化（DPO）存在的长度敏感性及冗长输出问题，提出LD-DPO方法，通过解耦显式长度偏好与隐式偏好实现长度脱敏，实验表明其在多个模型和基准上优于DPO，响应长度减少10-40%且更贴近人类偏好。
摘要：
直接偏好优化（DPO）在基于人类反馈的强化学习（RLHF）阶段被广泛用于使大语言模型（LLMs）与人类偏好对齐，从而提升模型的无害性和有效性。然而，研究发现DPO存在过度优化冗长性的问题，这会对性能和用户体验产生不利影响。本文对DPO的优化目标进行了深入的理论分析，揭示其隐式奖励与数据长度之间存在强相关性。这种相关性会误导优化方向，导致DPO训练过程中出现长度敏感性并产生冗长输出。为解决该问题，我们提出了一种DPO的长度脱敏改进方法，称为LD-DPO。该方法通过将相对次要的显式长度偏好与其他隐式偏好解耦，使DPO对数据长度脱敏，从而更有效地学习内在偏好。我们在Llama2-13B、Llama3-8B和Qwen2-7B的两种设置（Base和Instruct）上，在MT-Bench和AlpacaEval 2等多个基准上进行了实验验证。实验结果表明，LD-DPO持续优于DPO和其他基线方法，与DPO相比，响应长度减少了10-40%，且更简洁。我们通过深入的实验分析证明，LD-DPO确实能够实现长度脱敏，并使模型更贴近类人偏好。
英文摘要：
Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy. However, it has been observed that DPO tends to over-optimize for verbosity, which can detrimentally affect both performance and user experience. In this paper, we conduct an in-depth theoretical analysis of DPO's optimization objective and reveal a strong correlation between its implicit reward and data length. This correlation misguides the optimization direction, resulting in length sensitivity during the DPO training and leading to verbosity. To address this issue, we propose a length-desensitization improvement method for DPO, termed LD-DPO. The proposed method aims to desensitize DPO to data length by decoupling explicit length preference, which is relatively insignificant, from the other implicit preferences, thereby enabling more effective learning of the intrinsic preferences. We utilized two settings (Base and Instruct) of Llama2-13B, Llama3-8B, and Qwen2-7B for experimental validation on various benchmarks including MT-Bench and AlpacaEval 2. The experimental results indicate that LD-DPO consistently outperforms DPO and other baseline methods, achieving more concise responses with a 10-40% reduction in length compared to DPO. We conducted in-depth experimental analyses to demonstrate that LD-DPO can indeed achieve length desensitization and align the model more closely with human-like preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
