朝着解耦偏好优化动态:压制失败者,保留胜利者
Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
AI总结:
本文提出奖励校准方法,通过解耦带宽条件实现压制失败者并保留胜利者的优化动态,提升了下游性能。
摘要:
偏好优化被广泛用于对齐大语言模型(LLMs)与人类偏好。然而,许多基于边际的方法在尝试压制被拒绝响应时也会压制被选中的响应,且没有通用的方法能够在不同目标下防止这种情况。我们通过偏好优化的统一激励分数分解来解决这个问题,揭示了不同目标共享相同的局部更新方向,仅在标量权重上有所不同。这种分解为之前在单独设置中研究的目标提供了一个共同的分析框架。基于这种分解,通过分析所选/被拒绝似然的动力学,我们识别出解耦带宽(DB),一个简单且可测试的条件,它告诉我们何时训练可以遵循期望的路径:压制失败者,保留胜利者,可能在早期阶段之后。使用DB,我们提出了奖励校准(RC),一种即插即用的方法,能够自适应地重新平衡所选和被拒绝响应的更新,以满足DB,而无需重新设计基础目标。实验证果表明,RC导致了更解耦的动力学,在多个设置中观察到更好的下游性能。我们的代码可在https://github.com/IceyWuu/DisentangledPreferenceOptimization上获得。
英文摘要:
Preference optimization is widely used to align large language models (LLMs) with human preferences. However, many margin-based methods also suppress the chosen response when they try to suppress the rejected one, and there is no general way to prevent this across different objectives. We address this issue with a unified incentive-score decomposition of preference optimization, revealing that different objectives share the same local update directions and differ only in their scalar weights. This decomposition provides a common framework for analyzing objectives that were previously studied in separate settings. Building on this decomposition, by analyzing the dynamics of the chosen/rejected likelihoods, we identify the disentanglement band (DB), a simple, testable condition that tells us when training can follow the desired path: suppress the loser while preserving the winner, possibly after an early stage. Using the DB, we propose reward calibration (RC), a plug-and-play method that adaptively rebalances the updates for chosen and rejected responses to satisfy the DB, without redesigning the base objective. Empirical results show that RC leads to more disentangled dynamics, with better downstream performance observed across several settings. Our code is available at https://github.com/IceyWuu/DisentangledPreferenceOptimization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。