arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09824cs.CL
收藏

偏好噪声对生成式语言模型对齐性能的影响

Impact of Preference Noise on the Alignment Performance of Generative Language Models

Google Research(谷歌研究院)

Yang Gao, Dana Alon, Donald Metzler

AI总结:

本研究针对生成式语言模型对齐中20-40%偏好数据存在噪声的问题,提出可控噪声注入框架,系统验证噪声率对对齐性能的强负面影响,并证实基于置信度的数据过滤可有效缓解特定类型噪声的干扰。

摘要:

开发生成式语言模型(Generative Language Models,GLMs)的一项核心要求是使其价值观与人类价值观对齐。基于偏好的对齐是实现这一目标的广泛使用范式,该范式首先从人类标注者或AI系统中获取对生成文本对的偏好,再将其输入到Direct Preference Optimization(直接偏好优化)等对齐技术中。然而,GLM对齐中使用的偏好对有相当大比例(20-40%)存在噪声,目前尚不清楚这类噪声如何影响对齐性能,以及如何缓解其负面影响。本文提出了一个框架,可向偏好数据中注入可控量级和类型的噪声,并在摘要生成和对话生成两项任务中系统研究了偏好噪声对对齐性能的影响。研究发现,对齐性能对偏好数据中的噪声率高度敏感:例如,噪声率每提升10个百分点(pp),对齐性能(以胜率计)可能下降30个百分点。为缓解噪声影响,基于置信度的数据过滤在存在特定类型噪声时表现出显著效果。我们希望本研究能帮助学界更好地理解并缓解GLM对齐中偏好噪声的影响。

英文摘要:

A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affects the alignment performance and how to mitigate its negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑