arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.04102cs.LGcs.AIcs.CL
收藏

ROPO:大型语言模型的鲁棒偏好优化

ROPO: Robust Preference Optimization for Large Language Models

University of Science and Technology of China(中国科学技术大学);Alibaba Cloud(阿里云);HKUST(香港科技大学)

Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, Jieping Ye

AI总结:

ROPO提出一种迭代鲁棒偏好优化框架,通过动态权重和鲁棒损失过滤噪声样本,无需外部模型,显著提升LLM偏好对齐性能,且优势随噪声率增加而增强。

摘要:

偏好对齐对于赋能大型语言模型(LLMs)生成有用且无害的响应至关重要。然而,偏好对齐的性能对偏好数据中普遍存在的噪声高度敏感。针对此问题,近期的工作要么仅能轻微缓解噪声的影响,而无法真正减少噪声的存在,要么依赖成本高昂且易出现奖励误泛化的教师LLM。为应对这些挑战,我们提出了鲁棒偏好优化(ROPO)框架,这是一种无需外部模型辅助的迭代对齐方法,集成了噪声容忍与噪声样本过滤。具体而言,ROPO迭代求解一个约束优化问题,其中我们为每个样本动态分配一个质量感知权重,并将权重之和约束为我们打算保留的样本数量。为实现噪声容忍训练和有效的噪声识别,我们通过抑制高不确定性样本的梯度推导出一种鲁棒损失。我们从经验和理论上均证明,所推导的损失对于区分噪声样本与干净样本至关重要。此外,受我们所推导损失的启发,我们提出了一种鲁棒性引导的拒绝采样技术,以补偿被丢弃查询中可能的重要信息。在三个广泛使用的数据集上使用Mistral-7B和Llama-2-7B进行的实验表明,ROPO显著优于现有的偏好对齐方法,且其优势随噪声率的增加而增大。

英文摘要:

Preference alignment is pivotal for empowering large language models (LLMs) to generate helpful and harmless responses. However, the performance of preference alignment is highly sensitive to the prevalent noise in the preference data. Recent efforts for this problem either marginally alleviate the impact of noise without the ability to actually reduce its presence, or rely on costly teacher LLMs prone to reward misgeneralization. To address these challenges, we propose the RObust Preference Optimization (ROPO) framework, an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Specifically, ROPO iteratively solves a constrained optimization problem, where we dynamically assign a quality-aware weight for each sample and constrain the sum of the weights to the number of samples we intend to retain. For noise-tolerant training and effective noise identification, we derive a robust loss by suppressing the gradients of samples with high uncertainty. We demonstrate both empirically and theoretically that the derived loss is critical for distinguishing noisy samples from clean ones. Furthermore, inspired by our derived loss, we propose a robustness-guided rejection sampling technique to compensate for the potential important information in discarded queries. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods, with its superiority growing as the noise rate increases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑