传递性与循环性:动态大语言模型对齐的显式偏好分解
Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
AI总结:
本文提出Hybrid Reward-Cyclic模型,通过博弈论分解显式分离传递性和循环性偏好,结合动态自我博弈优化方法提升大语言模型对齐效果,实验证明其在混合传递-循环设置中具有结构优势和更高的准确率。
摘要:
标准的RLHF依赖于传递性的标量奖励,无法捕捉人类偏好的循环性质。尽管一些方法如通用偏好模型(GPM)试图解决这一问题,但其隐式公式将层次结构与循环性结合在一起,未能保证主导解。为此,我们提出了混合奖励-循环(HRC)模型,利用博弈论分解将偏好显式分解为正交的传递性(标量)和循环性(向量)组件。此外,我们引入了动态自我博弈偏好优化(DSPPO),将对齐视为随时间变化的游戏,逐步引导策略向纳什均衡发展。合成数据实验进一步验证了HRC在混合传递-循环设置中的结构优势,其中HRC收敛速度更快且准确率更高。在RewardBench 2上的实验表明,HRC在BT和GPM基线基础上持续改进(例如,在Gemma-2B-it上提升1.23%)。特别是,其在Ties领域中的优越表现验证了模型在处理复杂非严格偏好时的鲁棒性。对AlpacaEval 2.0、Arena-Hard-v0.1和MT-Bench的广泛下游评估确认了我们框架的有效性。值得注意的是,当使用Gemma-2B-it作为基础偏好模型时,HRC+DSPPO在AlpacaEval 2.0上达到峰值长度控制下的胜率44.75%,在Arena-Hard-v0.1上达到46.8%,显著优于使用BT或GPM训练的SPPO基线。我们的代码在https://github.com/lab-klc/Hybrid-Reward-Cyclic上公开可用。
英文摘要:
Standard RLHF relies on transitive scalar rewards, failing to capture the cyclic nature of human preferences. While some approaches like the General Preference Model (GPM) address this, we identify a theoretical limitation: their implicit formulation entangles hierarchy with cyclicity, failing to guarantee dominant solutions. To address this, we propose the Hybrid Reward-Cyclic (HRC) model, which utilizes game-theoretic decomposition to explicitly disentangle preferences into orthogonal transitive (scalar) and cyclic (vector) components. Complementing this, we introduce Dynamic Self-Play Preference Optimization (DSPPO), which treats alignment as a time-varying game to progressively guide the policy toward the Nash equilibrium. Synthetic data experiments further validate HRC's structural superiority in mixed transitive--cyclic settings, where HRC converges faster and achieves higher accuracy than GPM. Experiments on RewardBench 2 demonstrate that HRC consistently improves over both BT and GPM baselines (e.g., +1.23% on Gemma-2B-it). In particular, its superior performance in the Ties domain empirically validates the model's robustness in handling complex, non-strict preferences. Extensive downstream evaluations on AlpacaEval 2.0, Arena-Hard-v0.1, and MT-Bench confirm the efficacy of our framework. Notably, when using Gemma-2B-it as the base preference model, HRC+DSPPO achieves a peak length-controlled win-rate of 44.75% on AlpacaEval 2.0 and 46.8% on Arena-Hard-v0.1, significantly outperforming SPPO baselines trained with BT or GPM. Our code is publicly available at https://github.com/lab-klc/Hybrid-Reward-Cyclic.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。