arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07327cs.AIcs.CLcs.LG
收藏

3D-Properties:识别DPO中的挑战并规划前进路径

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Tsinghua University(清华大学);Qing Yuan Research Institute(清源研究院);Shanghai Jiao Tong University(上海交通大学);Baichuan AI(百川智能)

Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, Dong Yan

AI总结:

本文重新审视DPO,识别出被拒绝响应似然急剧下降、退化为响应抑制和对未见响应的分散效应三个关键问题(3D性质),分析其优化动力学根源,并提出正则化技术以提升训练稳定性与性能。

摘要:

将大语言模型(LLMs)与人类偏好对齐已受到广泛关注,其中近端策略优化(PPO)是标准但计算成本高昂的方法,而直接偏好优化(DPO)则是更高效的替代方案。尽管DPO实现简单,但它在最先进的大语言模型中仍未得到充分利用,这表明其可能存在局限性。在本工作中,我们重新审视DPO,分析其理论基础和实证表现以弥合这一差距。我们识别出DPO学习过程中出现的三个关键性质,统称为3D性质:被拒绝响应似然的急剧下降(Drastic drop)、退化为响应抑制(Degradation into response suppression)以及对未见响应的分散效应(Dispersion effect)。我们证明这些问题源于DPO的优化动力学,其中被选择响应和被拒绝响应的梯度之间的相互作用导致不稳定性。我们的发现得到了受控玩具模型和真实世界大语言模型任务实验的支持,包括数学问题求解和指令遵循。为应对这些挑战,我们提出了简单的正则化技术,以提高训练稳定性和性能。此外,我们考察了偏好数据分布如何影响DPO的有效性,为对齐模型如何处理域外(OOD)数据提供了见解。我们的工作将这些观察结果与更广泛的研究联系起来,并为DPO的局限性提供了理论解释。我们希望这些见解能够指导无奖励模型偏好学习的未来发展,使其更接近基于奖励模型的方法。

英文摘要:

Aligning large language models (LLMs) with human preferences has gained significant attention, with Proximal Policy Optimization (PPO) as a standard yet computationally expensive method and Direct Preference Optimization (DPO) as a more efficient alternative. While DPO offers simplicity, it remains underutilized in state-of-the-art LLMs, suggesting potential limitations. In this work, we revisit DPO, analyzing its theoretical foundations and empirical performance to bridge this gap. We identify three key properties, termed 3D properties, that emerge from DPO's learning process: Drastic drop in rejected response likelihood, Degradation into response suppression, and Dispersion effect on unseen responses. We show that these issues arise from DPO's optimization dynamics, where the interaction between chosen and rejected response gradients leads to instability. Our findings are supported by experiments on both a controlled toy model and real-world LLM tasks, including mathematical problem-solving and instruction following. To address these challenges, we propose simple regularization techniques that improve training stability and performance. Additionally, we examine how preference data distribution impacts DPO's effectiveness, offering insights into how alignment models handle out-of-domain (OOD) data. Our work connects these observations to broader research and provides a theoretical explanation for DPO's limitations. We hope these insights will guide future advancements in reward-model-free preference learning, bringing it closer to reward-model-based approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑