BPO：重新审视直接偏好优化中的偏好建模
BPO: Revisiting Preference Modeling in Direct Preference Optimization
AI总结：
本文提出平衡偏好优化（BPO），通过平衡奖励边际和差距适配器动态平衡已选与拒绝响应的优化，从根本上解决DPO的已选响应退化问题，在数学推理任务上显著提升准确率且仅需一行代码修改。
摘要：
直接偏好优化（DPO）已成为将大语言模型（LLMs）与人类偏好对齐的流行方法。虽然DPO通过成对排序损失有效保持了已选响应与拒绝响应之间的相对顺序，但它常常忽略绝对奖励幅度。这一疏忽可能降低已选响应的似然，并增加生成分布外响应的风险，从而导致性能下降。我们将这一问题称为“已选响应退化”（Degraded Chosen Responses，DCR）。为解决这一问题，我们提出平衡偏好优化（Balanced Preference Optimization，BPO），这是一个新颖的框架，通过两个关键组件——平衡奖励边际（balanced reward margin）和差距适配器（gap adaptor）——动态平衡已选响应与拒绝响应的优化。与以往方法不同，BPO能够从根本上解决DPO的DCR问题，而无需向损失函数引入额外约束。在多个数学推理任务上的实验结果表明，BPO显著优于DPO，使用Llama-3.1-8B-Instruct时准确率提升+10.1%（从18.8%到28.9%），使用Qwen2.5-Math-7B时提升+11.7%（从35.0%到46.7%）。在同一模型上，它还分别超过DPO变体IPO（43.1%）+3.6%、SLiC（41.7%）+5.0%和Cal-DPO（43.6%）+3.1%。值得注意的是，我们的算法只需修改一行代码，实现简单，并且与现有基于DPO的框架完全兼容。
英文摘要：
Direct Preference Optimization (DPO) have emerged as a popular method for aligning Large Language Models (LLMs) with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR).To address this issue, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: balanced reward margin and gap adaptor. Unlike previous methods, BPO can fundamentally resolve DPO's DCR issue, without introducing additional constraints to the loss function. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct (18.8% to 28.9%) and +11.7% with Qwen2.5-Math-7B (35.0% to 46.7%). It also surpasses DPO variants by +3.6% over IPO (43.1%), +5.0% over SLiC (41.7%), and +3.1% over Cal-DPO (43.6%) on the same model. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
