同时进行奖励蒸馏与偏好学习：获得一个能兼顾两者的语言模型
Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both
AI总结：
本文提出 DRDO 方法，通过同时进行奖励蒸馏与偏好学习，克服了 DPO 等直接对齐方法在处理噪声偏好标签时导致的策略退化问题，在期望奖励和鲁棒性上均表现更优。
摘要：
传统的基于RLHF的LLM对齐方法显式地最大化来自独立奖励模型的期望奖励。更近期的监督对齐方法如 Direct Preference Optimization (DPO) 规避了这一阶段，以避免模型漂移和奖励过拟合等问题。尽管因其简单性而广受欢迎，但DPO及类似的重度依赖基于 Bradley-Terry 的成对偏好建模的直接对齐方法，在面对非确定性或带噪声的偏好标签（例如人类对两个候选输出以低置信度进行评分）时，仍可能导致策略退化。本文提出了 DRDO (Direct Reward Distillation and policy-Optimization)，该方法同时建模奖励和偏好以避免此类退化。DRDO 在通过一种新颖的偏好似然公式学习人类偏好的同时，直接模拟由 oracle 分配的奖励。在 Ultrafeedback 和 TL;DR 数据集上的结果表明，经 DRDO 训练的策略在期望奖励方面超越了 DPO 和 e-DPO 等方法，并且平均而言，对带噪声的偏好信号以及分布外 (OOD) 设置具有更强的鲁棒性。
英文摘要：
Traditional RLHF-based LLM alignment methods explicitly maximize the expected rewards from a separate reward model. More recent supervised alignment methods like Direct Preference Optimization (DPO) circumvent this phase to avoid problems including model drift and reward overfitting. Although popular due to its simplicity, DPO and similar direct alignment methods which rely heavily on the Bradley-Terry-based pairwise preference formulation can still lead to degenerate policies when challenged by non-deterministic or noisy preference labels, for example human scoring of two candidate outputs with low confidence. This paper introduces DRDO (Direct Reward Distillation and policy-Optimization), which simultaneously models rewards and preferences to avoid such degeneracy. DRDO directly mimics rewards assigned by an oracle while learning human preferences with a novel preference likelihood formulation. Results on the Ultrafeedback and TL;DR datasets demonstrate that DRDO-trained policies surpass methods such as DPO and e-DPO in terms of expected rewards and are more robust, on average, to noisy preference signals as well as out-of-distribution (OOD) settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
