arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.20847cs.LG
收藏

直接偏好优化中的梯度不平衡

Gradient Imbalance in Direct Preference Optimization

Rutgers University(罗格斯大学);IIIS, Tsinghua University(清华大学交叉信息研究院);University of Washington(华盛顿大学);University of Copenhagen(哥本哈根大学)

Qinwei Ma, Jingzhe Shi, Can Jin, Jenq-Neng Hwang, Serge Belongie, Lei Li

AI总结:

本文系统分析直接偏好优化(DPO)的训练动态,发现梯度不平衡导致性能欠佳,并提出Balanced-DPO通过计算高效的梯度重加权机制解决此问题,实验验证其有效性,为提升DPO性能指明方向。

摘要:

直接偏好优化(DPO)已被提出作为基于近端策略优化(PPO)的强化学习与人类反馈(RLHF)的一种有前景的替代方案。然而,实证评估一致显示,与常见的RLHF流程相比,DPO的性能欠佳。在本工作中,我们对DPO的训练动态进行了系统分析,并识别出梯度不平衡是一个关键限制。我们在理论上和实证上证明,这种不平衡会扰动优化轨迹、破坏学习稳定性,并导致次优收敛。为解决此问题,我们提出了Balanced-DPO,一种对DPO目标函数的简单而有效的修改,引入了一种计算高效的梯度重新加权机制。我们的实验证明了Balanced-DPO的有效性,验证了理论发现,并确认解决梯度不平衡是提升DPO性能的关键,为未来研究指明了有前景的方向。

英文摘要:

Direct Preference Optimization (DPO) has been proposed as a promising alternative to Proximal Policy Optimization (PPO) based Reinforcement Learning with Human Feedback (RLHF). However, empirical evaluations consistently reveal suboptimal performance in DPO compared to common RLHF pipelines. In this work, we conduct a systematic analysis of DPO's training dynamics and identify gradient imbalance as a critical limitation. We demonstrate theoretically and empirically that this imbalance perturbs optimization trajectories, destabilizes learning, and induces suboptimal convergence. To address this issue, we propose Balanced-DPO, a simple yet effective modification to the DPO objective that introduces a computationally efficient gradient reweighting mechanism. Our experiments demonstrate the effectiveness of Balanced-DPO, validating the theoretical findings and confirming that addressing gradient imbalance is key to improving DPO's performance, highlighting a promising direction for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑