反思DPO：被拒响应在偏好错位中的作用
Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
AI总结：
针对DPO因被拒响应主导损失函数而偏好优化失衡的问题，提出保留原结构同时约束被拒响应影响的BDPO方法，经理论与实验验证其优化更均衡、性能更优。
摘要：
直接偏好优化（Direct Preference Optimization, DPO）是一种简洁高效的框架，已受到广泛关注。然而，由于被拒响应对损失函数的主导影响，它往往难以实现其核心目标——提升选中响应的生成概率，同时降低被拒响应的生成概率。这种失衡导致其在推广偏好响应方面的性能欠佳。本研究系统分析了DPO及现有旨在实现上述目标的算法的局限性。为解决这些局限，我们提出了有界DPO（Bounded-DPO, BDPO），这是一种在保留DPO原有优化结构的同时，对被拒响应的影响加以约束的新方法。通过理论分析和实证评估，我们证明BDPO实现了选中响应与被拒响应的均衡优化，性能优于现有算法。
英文摘要：
Direct Preference Optimization (DPO) is a simple and efficient framework that has attracted substantial attention. However, it often struggles to meet its primary objectives -- increasing the generation probability of chosen responses while reducing that of rejected responses -- due to the dominant influence of rejected responses on the loss function. This imbalance leads to suboptimal performance in promoting preferred responses. In this work, we systematically analyze the limitations of DPO and existing algorithms designed to achieve the objectives stated above. To address these limitations, we propose Bounded-DPO (BDPO), a novel method that bounds the influence of rejected responses while maintaining the original optimization structure of DPO. Through theoretical analysis and empirical evaluations, we demonstrate that BDPO achieves a balanced optimization of the chosen and rejected responses, outperforming existing algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
