对抗位移的DPO扩展:非凸f-散度
Displacement-Resistant Extensions of DPO with Nonconvex $f$-Divergences
AI总结:
本文提出了一种对抗位移的DPO扩展方法,通过非凸f-散度改进了传统DPO,提供了更强的理论保证和实际表现。
摘要:
DPO及相关算法通过直接优化RLHF目标对语言模型进行对齐:寻找一个在KL散度惩罚下接近参考策略的策略,以最大化布拉德-蒂尔利奖励。先前的工作表明,这种方法可以进一步推广:即使将KL散度替换为具有凸生成函数f的f-散度家族,原始问题仍然保持可处理。我们的第一个贡献是证明f的凸性不是必需的。相反,我们识别了一个更一般的条件,称为DPO诱导,它精确地表征了何时RLHF问题仍然可处理。我们的下一个贡献是建立一个关于f的第二个条件,该条件必要地防止位移,这是已知的实验证据现象,其中赢家和输家响应的概率接近零。我们称满足此条件的任何f为位移抗性。最后,我们专注于一个特定的DPO诱导和位移抗性f,导致我们新的SquaredPO损失。与DPO相比,这种新损失在理论上提供了更强的保证,同时在实践中表现具有竞争力。
英文摘要:
DPO and related algorithms align language models by directly optimizing the RLHF objective: find a policy that maximizes the Bradley-Terry reward while staying close to a reference policy through a KL divergence penalty. Previous work showed that this approach could be further generalized: the original problem remains tractable even if the KL divergence is replaced by a family of $f$-divergence with a convex generating function $f$. Our first contribution is to show that convexity of $f$ is not essential. Instead, we identify a more general condition, referred to as DPO-inducing, that precisely characterizes when the RLHF problem remains tractable. Our next contribution is to establish a second condition on $f$ that is necessary to prevent probability displacement, a known empirical phenomenon in which the probabilities of the winner and the loser responses approach zero. We refer to any $f$ that satisfies this condition as displacement-resistant. We finally focus on a specific DPO-inducing and displacement-resistant $f$, leading to our novel SquaredPO loss. Compared to DPO, this new loss offers stronger theoretical guarantees while performing competitively in practice.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。