arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00954cs.AI
收藏

小边距偏好依然重要-如果你训练得当的话

Small-Margin Preferences Still Matter-If You Train Them Right

University of California, Santa Cruz(加州大学圣克鲁兹分校);Northeastern University(东北大学);DIMACS, Rutgers University(Rutgers大学DIMACS研究中心)

Jinlong Pang, Zhaowei Zhu, Na Di, Yichi Zhang, Yaxuan Wang, Chen Qian, Yang Liu

AI总结:

本文提出MixDPO,一种基于难度的训练策略,通过课程学习和混合目标优化,提升LLM在模糊配对上的对齐效果。

摘要:

偏好优化方法,如DPO,通过成对比较对大型语言模型(LLMs)进行对齐,但其效果可能对偏好对的质量和难度非常敏感。一种常见的启发式方法将小边距(模糊)对视为噪声并过滤掉它们。在本文中,我们重新审视这一假设,并表明配对难度与优化目标之间存在强烈的相互作用:当使用基于偏好的损失进行训练时,困难的配对可能会破坏训练并损害对齐,然而这些相同的配对在使用监督微调(SFT)进行优化时仍然包含有用的监督信号。受这一观察的启发,我们提出了MixDPO,一种简单而有效的难度意识训练策略,它(i)按从易到难的顺序对偏好数据进行排序(一种基于边距定义难度的课程学习),并且(ii)将困难配对路由到SFT目标,同时对容易的配对应用偏好损失。这种混合设计提供了一种实用的机制,可以在不导致与低边距数据上的偏好损失相关的优化失败的情况下利用模糊配对。在三个LLM-judge基准测试中,MixDPO在DPO和一系列广泛使用的变体上始终提高了对齐效果,特别是在AlpacaEval~2长度控制(LC)胜利率上表现尤为突出。

英文摘要:

Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficulty of preference pairs. A common heuristic treats small-margin (ambiguous) pairs as noisy and filters them out. In this paper, we revisit this assumption and show that pair difficulty interacts strongly with the optimization objective: when trained with preference-based losses, difficult pairs can destabilize training and harm alignment, yet these same pairs still contain useful supervision signals when optimized with supervised fine-tuning (SFT). Motivated by this observation, we propose MixDPO, a simple yet effective difficulty-aware training strategy that (i) orders preference data from easy to hard (a curriculum over margin-defined difficulty), and (ii) routes difficult pairs to an SFT objective while applying a preference loss to easy pairs. This hybrid design provides a practical mechanism to leverage ambiguous pairs without incurring the optimization failures often associated with preference losses on low-margin data. Across three LLM-judge benchmarks, MixDPO consistently improves alignment over DPO and a range of widely-used variants, with particularly strong gains on AlpacaEval~2 length-controlled (LC) win rate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑