基于人类反馈的强化学习(Reinforcement learning from human feedback, RLHF)为使AI系统对齐人类偏好数据提供了一套原则性框架。由于个人偏见、语境模糊、标注员缺乏培训等多种原因,人类标注者可能给出错误或不一致的偏好标签。为应对这一挑战,我们提出一种鲁棒RLHF方法——$R^3M$,该方法将可能受污染的偏好标签建模为稀疏异常值。相应地,我们将鲁棒奖励学习建模为一个$\boldsymbol{\rm \u2113}_1$正则化的最大似然估计问题。在计算层面,我们开发了一种高效的交替优化算法,与标准RLHF方法相比,其仅产生可忽略的计算开销。在理论层面,我们证明在适当的正则性条件下,只要异常值标签的数量与偏好样本量呈次线性增长关系,$R^3M$就能一致地学习到潜在奖励并识别出异常值。此外,我们指出$R^3M$具备通用性,可扩展至多种偏好优化方法,包括直接偏好优化(direct preference optimization, DPO)。我们在机器人控制以及大语言模型(large language models, LLMs)的自然语言生成任务上开展的实验表明,$R^3M$提升了奖励模型针对偏好数据多种扰动类型的鲁棒性。
英文摘要:
Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels. To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an $\ell_1$-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, $R^3M$ can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that $R^3M$ is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R^3M$ improves robustness of the reward against several types of perturbations to the preference data.
ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学深圳分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing(计算机科学与技术系,人工智能研究院,BNRist中心,清华大学-博世联合机器学习中心,THBI实验室,清华大学,北京);Dept. of Automation, Tsinghua University, Beijing(自动化系,清华大学,北京)
ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback
Department of Computer Science, Cleveland State University(克利夫兰州立大学计算机科学系);School of Computing and Information Sciences, Florida International University(佛罗里达国际大学计算与信息科学学院);Data Science and Learning Division, Argonne National Laboratory(阿贡国家实验室数据科学与学习分部)
Strategyproof Reinforcement Learning from Human Feedback
ETH Zurich(苏黎世联邦理工学院);ETH AI Center(ETH人工智能中心);Department of Computer Science University of Oxford(牛津大学计算机科学系);Department of Computer Science University of Warwick(Warwick大学计算机科学系)
Reinforcement Learning with Segment Reward Feedback under Linear Function Approximation
National University of Singapore(新加坡国立大学);Microsoft Research Asia(微软亚洲研究院);NVIDIA Research(英伟达研究院);Technion(以色列理工学院);Singapore University of Technology and Design(新加坡科技设计大学)
Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);City University of Hong Kong(香港城市大学);Rice University(莱斯大学);University of Liverpool(利物浦大学);Tencent Robotics X(腾讯Robotics X实验室)
Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
Department of Statistics and Data Science, Chinese University of Hong Kong(统计与数据科学系,香港中文大学);Department of Statistics and Data Science, the Wharton School, University of Pennsylvania(统计与数据科学系,沃顿商学院,宾夕法尼亚大学)
Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
Department of Statistics and Operations Research, University of North Carolina, Chapel Hill(统计与运筹学系,北卡罗来纳大学 Chapel Hill 分校);Daniels School of Business, Purdue University(商务学院,普渡大学);Department of Statistics and Operations Research, Department of Genetics, Department of Biostatistics, The University of North Carolina at Chapel Hill(统计与运筹学系,遗传学系,生物统计学系,北卡罗来纳大学 Chapel Hill 分校)
Reinforcement Learning from Human Feedback: A Statistical Perspective
Department of Biostatistics, Yale University(耶鲁大学生物统计学系);Department of Statistics, London School of Economics and Political Science(伦敦政治经济学院统计学系);Mitch Daniels School of Business, Purdue University(普渡大学米奇·丹尼尔斯商学院)