arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.05342cs.LGcs.AI
收藏

Margin Adaptive DPO: 利用奖励模型实现偏好优化中的细粒度控制

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

Independent Researcher(独立研究者)

Hyung Gyu Rho

AI总结:

提出Margin-Adaptive Direct Preference Optimization (MADPO)方法,通过奖励模型估计偏好边界并自适应调整DPO损失权重,实现实例级别的细粒度控制,在摘要任务上优于现有方法。

摘要:

直接偏好优化(DPO)已成为一种简单有效的大语言模型对齐方法。然而,其依赖固定温度参数导致在多样化偏好数据上训练次优,造成对简单样本过拟合而对信息丰富样本学习不足。近期出现了应对此问题的方法。虽然IPO解决了通用过拟合,但其均匀正则化可能过于保守。更针对性的β-DPO方法有其自身局限:其批次级自适应对混合边界对应用单一折中温度,线性更新规则可能产生不稳定的负β值,其过滤机制丢弃了潜在有用的训练信号。本文提出边界自适应直接偏好优化(MADPO),一种稳定、保留数据且实例级别的解决方案。MADPO采用实用的两步方法:首先训练奖励模型估计偏好边界,然后利用这些边界对每个训练样本的DPO损失施加连续自适应权重。这种重加权方案创建了一个有效目标边界,对困难对放大而对简单对抑制,从而实现对学习信号的细粒度控制。我们提供了全面的理论分析,证明MADPO具有良性的优化景观,且对奖励模型估计误差具有鲁棒性。我们通过使用人类偏好数据的摘要任务实验验证了理论。MADPO在全面的解码温度扫描中一致优于强基线。

英文摘要:

Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models. However, its reliance on a fixed temperature parameter leads to suboptimal training on diverse preference data, causing overfitting on easy examples and under-learning from informative ones. Recent methods have emerged to counter this. While IPO addresses general overfitting, its uniform regularization can be overly conservative. The more targeted approach of $β$-DPO suffers from its own limitations: its batch-level adaptation applies a single, compromised temperature to mixed-margin pairs, its linear update rule can produce unstable negative $β$ values, and its filtering mechanism discards potentially useful training signals. In this work, we introduce Margin-Adaptive Direct Preference Optimization (MADPO), a method that provides a stable, data-preserving, and instance-level solution. MADPO employs a practical two-step approach: it first trains a reward model to estimate preference margins and then uses these margins to apply a continuous, adaptive weight to the DPO loss for each individual training sample. This re-weighting scheme creates an effective target margin that is amplified for hard pairs and dampened for easy pairs, allowing for granular control over the learning signal. We provide a comprehensive theoretical analysis, proving that MADPO has a well-behaved optimization landscape and is robust to reward model estimation errors. We validate our theory with experiments on a summarization task using human preference data. MADPO consistently outperforms strong baselines across a comprehensive sweep of decoding temperatures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑