arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.14643cs.CL
收藏

无参考模型的长度可控基于间隔的偏好优化

Length-Controlled Margin-Based Preference Optimization without Reference Model

School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)

Gengxu Li, Tingyu Xia, Yi Chang, Yuan Wu

AI总结:

本研究针对DPO的局限提出LMPO,引入统一参考模型与平均对数概率优化策略,通过长度可控间隔损失调控响应长度并缓解概率退化,在Mistral和LLaMA3上的6个基准中性能优于现有方法。

摘要:

直接偏好优化(DPO)是一种广泛应用于基于人类反馈的偏好强化学习(RLHF)的离线算法,旨在通过重新定义奖励函数提升训练的简洁性与稳定性。然而DPO存在长度偏差、内存效率低、概率退化等局限。为解决这些挑战,我们提出更高效鲁棒的替代方法:长度可控基于间隔的偏好优化(LMPO)。LMPO引入统一参考模型作为DPO损失的上界,能更精准近似原始优化目标;同时采用平均对数概率优化策略,最小化训练与推理阶段的差异。LMPO的核心创新是在Bradley-Terry框架内整合的长度可控基于间隔的损失函数,该函数可调控响应长度,同时扩大偏好输出与拒绝输出间的间隔,缓解接受与丢弃响应的概率退化问题,解决了现有方法的重大局限。我们在两个开放端大型语言模型Mistral和LLaMA3上,针对6个条件基准,将LMPO与当前最优的偏好优化技术进行评估。实验结果表明,LMPO可有效控制响应长度、降低概率退化,且性能优于现有方法。代码可在指定URL获取。

英文摘要:

Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF), designed to improve training simplicity and stability by redefining reward functions. However, DPO is hindered by several limitations, including length bias, memory inefficiency, and probability degradation. To address these challenges, we propose Length-Controlled Margin-Based Preference Optimization (LMPO), a more efficient and robust alternative. LMPO introduces a uniform reference model as an upper bound for the DPO loss, enabling a more accurate approximation of the original optimization objective. Additionally, an average log-probability optimization strategy is employed to minimize discrepancies between training and inference phases. A key innovation of LMPO lies in its Length-Controlled Margin-Based loss function, integrated within the Bradley-Terry framework. This loss function regulates response length while simultaneously widening the margin between preferred and rejected outputs. By doing so, it mitigates probability degradation for both accepted and discarded responses, addressing a significant limitation of existing methods. We evaluate LMPO against state-of-the-art preference optimization techniques on two open-ended large language models, Mistral and LLaMA3, across six conditional benchmarks. Our experimental results demonstrate that LMPO effectively controls response length, reduces probability degradation, and outperforms existing approaches. The code is available at https://github.com/gengxuli/LMPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑