arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15568cs.LG
收藏

基于受污染人类反馈的鲁棒强化学习

Robust Reinforcement Learning from Corrupted Human Feedback

Georgia Tech(佐治亚理工学院);Amazon(亚马逊公司)

Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, Tuo Zhao

AI总结:

针对人类反馈偏好标签易受污染的问题,提出鲁棒RLHF方法$R^3M$,通过ℓ₁正则化建模稀疏异常值,兼具高效性与理论保证,可扩展至多类偏好优化方法,在机器人控制与大语言模型生成任务上验证了奖励鲁棒性的提升。

摘要:

基于人类反馈的强化学习(Reinforcement learning from human feedback, RLHF)为使AI系统对齐人类偏好数据提供了一套原则性框架。由于个人偏见、语境模糊、标注员缺乏培训等多种原因,人类标注者可能给出错误或不一致的偏好标签。为应对这一挑战,我们提出一种鲁棒RLHF方法——$R^3M$,该方法将可能受污染的偏好标签建模为稀疏异常值。相应地,我们将鲁棒奖励学习建模为一个$\boldsymbol{\rm \u2113}_1$正则化的最大似然估计问题。在计算层面,我们开发了一种高效的交替优化算法,与标准RLHF方法相比,其仅产生可忽略的计算开销。在理论层面,我们证明在适当的正则性条件下,只要异常值标签的数量与偏好样本量呈次线性增长关系,$R^3M$就能一致地学习到潜在奖励并识别出异常值。此外,我们指出$R^3M$具备通用性,可扩展至多种偏好优化方法,包括直接偏好优化(direct preference optimization, DPO)。我们在机器人控制以及大语言模型(large language models, LLMs)的自然语言生成任务上开展的实验表明,$R^3M$提升了奖励模型针对偏好数据多种扰动类型的鲁棒性。

英文摘要:

Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels. To tackle this challenge, we propose a robust RLHF approach -- $R^3M$, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an $\ell_1$-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, $R^3M$ can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that $R^3M$ is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R^3M$ improves robustness of the reward against several types of perturbations to the preference data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑