arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.13862cs.LGcs.CL
收藏

基于能量的偏好模型比 Bradley-Terry 偏好模型提供更好的离线对齐效果

Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

Yuzhong Hong, Hanshan Zhang, Junwei Bao, Hongfei Jiang, Yang Song

AI总结:

针对 DPO 因 Bradley-Terry 模型缺乏唯一最大似然估计导致 RLHF 损失最优解不可达的问题,提出具有唯一 MLE 的能量模型及对比损失 EPA,在开放基准上取得更优的离线对齐性能。

摘要:

自 DPO 首次亮相以来,已有研究表明,通过 KL 约束的 RLHF 损失将目标 LLM 与人类偏好对齐,在数学上等价于一种特殊的奖励建模任务。具体而言,该任务要求:1) 使用目标 LLM 来参数化奖励模型;2) 调整奖励模型,使其与真实奖励具有 1:1 的线性关系。然而,我们发现了一个重大问题:DPO 损失可能有多个最小化解,其中只有一个满足所需的线性条件。该问题源于其底层的 Bradley-Terry 偏好模型的一个众所周知的问题:它并不总是具有唯一的最大似然估计(MLE)。因此,RLHF 损失的最小化解可能是无法达到的,因为它仅仅是 DPO 损失众多最小化解中的一个。作为更好的替代方案,我们提出了一种基于能量的模型(EBM),它始终具有唯一的 MLE,且本质上满足线性要求。为了在实践中近似 MLE,我们提出了一种名为 Energy Preference Alignment (EPA) 的对比损失,其中每个正样本都会与一个或多个强负样本以及许多自由的弱负样本进行对比。我们的 EBM 的理论特性使得 EPA 的近似误差在使用了足够数量的负样本时几乎必然消失。在实验中,我们证明与 DPO 相比,EPA 在开放基准测试上始终提供更好的性能,从而展示了我们 EBM 的优越性。

英文摘要:

Since the debut of DPO, it has been shown that aligning a target LLM with human preferences via the KL-constrained RLHF loss is mathematically equivalent to a special kind of reward modeling task. Concretely, the task requires: 1) using the target LLM to parameterize the reward model, and 2) tuning the reward model so that it has a 1:1 linear relationship with the true reward. However, we identify a significant issue: the DPO loss might have multiple minimizers, of which only one satisfies the required linearity condition. The problem arises from a well-known issue of the underlying Bradley-Terry preference model: it does not always have a unique maximum likelihood estimator (MLE). Consequently,the minimizer of the RLHF loss might be unattainable because it is merely one among many minimizers of the DPO loss. As a better alternative, we propose an energy-based model (EBM) that always has a unique MLE, inherently satisfying the linearity requirement. To approximate the MLE in practice, we propose a contrastive loss named Energy Preference Alignment (EPA), wherein each positive sample is contrasted against one or more strong negatives as well as many free weak negatives. Theoretical properties of our EBM enable the approximation error of EPA to almost surely vanish when a sufficient number of negatives are used. Empirically, we demonstrate that EPA consistently delivers better performance on open benchmarks compared to DPO, thereby showing the superiority of our EBM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑