开发生成式语言模型(Generative Language Models,GLMs)的一项核心要求是使其价值观与人类价值观对齐。基于偏好的对齐是实现这一目标的广泛使用范式,该范式首先从人类标注者或AI系统中获取对生成文本对的偏好,再将其输入到Direct Preference Optimization(直接偏好优化)等对齐技术中。然而,GLM对齐中使用的偏好对有相当大比例(20-40%)存在噪声,目前尚不清楚这类噪声如何影响对齐性能,以及如何缓解其负面影响。本文提出了一个框架,可向偏好数据中注入可控量级和类型的噪声,并在摘要生成和对话生成两项任务中系统研究了偏好噪声对对齐性能的影响。研究发现,对齐性能对偏好数据中的噪声率高度敏感:例如,噪声率每提升10个百分点(pp),对齐性能(以胜率计)可能下降30个百分点。为缓解噪声影响,基于置信度的数据过滤在存在特定类型噪声时表现出显著效果。我们希望本研究能帮助学界更好地理解并缓解GLM对齐中偏好噪声的影响。
英文摘要:
A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affects the alignment performance and how to mitigate its negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
Harbin Institute of Technology(哈尔滨工业大学);Beijing Normal University - Hong Kong Baptist University United International College(北京师范大学-香港浸会大学联合国际学院);Qilu University of Technology(齐鲁工业大学)
A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
University of California, San Diego(加利福尼亚大学圣地亚哥分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Adobe Research(奥多比研究院);The Ohio State University(俄亥俄州立大学);Allen Institute for AI(艾伦人工智能研究所)
A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment
Soochow University(苏州大学);The Hong Kong Polytechnic University(香港理工大学);Beijing Jinghang Research Institute of Computing and Communication(北京京航计算通讯研究所);Nanyang Technological University(南洋理工大学)
Multi-Objective Exploration and Preference Optimization via Mutual Information
School of Computer, Beihang University(北京航空航天大学计算机学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰AGI实验室,中国电信人工智能技术(北京)有限公司)
Panacea: Pareto Alignment via Preference Adaptation for LLMs
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);BIGAI(北京智源人工智能研究院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Yuanpei College, Peking University(北京大学元培学院)
Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
Artefact Research Center(Artefact 研究中心);Unbabel;Equall;MICS, CentraleSupélec, Université Paris-Saclay(巴黎-萨克雷大学中央理工-高等电力学院数学与计算机科学实验室);Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico & Universidade de Lisboa(里斯本大学高等技术学院)
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
School of Computer Science and Engineering, Northeastern University(计算机科学与工程学院,东北大学);NiuTrans Research(NiuTrans研究院);CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS(行为科学重点实验室,心理学研究所)
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用关键实验室);Ministry of Education(教育部)
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Communication Content Cognition(通信内容认知国家重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Tencent Inc.(腾讯公司)