可证明鲁棒的DPO：利用含噪反馈对齐语言模型
Provably Robust DPO: Aligning Language Models with Noisy Feedback
AI总结：
针对含噪偏好数据影响DPO算法对齐效果的问题，提出带理论保证的鲁棒DPO框架，通过去偏损失函数实现噪声鲁棒性，实验验证其优于基线方法。
摘要：
近年来，基于偏好反馈的学习作为一种使语言模型对齐人类意图的有前景方法受到广泛关注。尽管这些对齐后的生成模型在各类任务中展现出令人瞩目的能力，但它们对高质量人类偏好数据的依赖成为实际应用中的瓶颈。具体而言，数据集中存在的噪声（错误且模棱两可的）偏好对可能限制语言模型准确捕捉人类意图。尽管从业者近期提出了一些启发式方法来缓解噪声偏好的影响，但人们对其工作原理仍缺乏完整的理论理解。\n在本研究中，我们旨在弥补这一空白，提出了一个存在随机偏好翻转情况下的策略优化通用框架。我们特别关注直接偏好优化（Direct Preference Optimization, DPO）算法，因为该算法假设偏好符合Bradley-Terry-Luce（BTL）模型，这引发了人们对噪声数据对所学策略影响的担忧。我们设计了一种新颖的损失函数，该函数可在平均意义上去除噪声的影响，使得通过最小化该损失训练的策略对噪声具有鲁棒性。在策略类的对数线性参数化且假设监督微调（Supervised Fine-Tuning, SFT）策略具有良好特征覆盖的条件下，我们证明了所提出的鲁棒DPO（robust DPO, rDPO）策略与最优策略相比的次优性差距为$O(\frac{1}{1-2ε}\sqrt{\frac{d}{n}})$，其中$ε< 1/2$为标签翻转率，$d$为策略参数维度，$n$为数据集规模。我们在IMDb情感生成任务和Anthropic的helpful-harmless数据集上的实验表明，与原始DPO及从业者提出的其他启发式方法相比，rDPO对偏好标签中的噪声具有鲁棒性。
英文摘要：
Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2ε}\sqrt{\frac{d}{n}})$, where $ε< 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
