arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.06080cs.AI
收藏

大型语言模型中RLHF的秘密(第二部分):奖励建模

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Hikvision Inc(海康威视);Fudan NLP Lab(复旦大学自然语言处理实验室);Fudan Vision and Learning Lab(复旦大学视觉与学习实验室)
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran F… 展开作者

Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang

AI总结:

本文针对RLHF中奖励模型面临的偏好数据噪声和分布外泛化问题,提出基于多模型投票的偏好强度度量与数据优化方法,并引入对比学习和元学习提升奖励模型的区分能力与迭代RLHF适用性。

摘要:

基于人类反馈的强化学习(RLHF)已成为使语言模型与人类价值观和意图对齐的关键技术,使模型能够产生更有帮助且更无害的回复。奖励模型被训练为人类偏好的代理,以驱动强化学习优化。尽管奖励模型通常被认为是实现高性能的核心,但它们在实际应用中面临以下挑战:(1)数据集中不正确和模糊的偏好对可能阻碍奖励模型准确捕捉人类意图。(2)在特定分布数据上训练的奖励模型通常难以泛化到该分布之外的样本,并且不适合迭代式RLHF训练。在本报告中,我们尝试解决这两个问题。(1)从数据角度,我们提出了一种基于多个奖励模型投票机制的方法来衡量数据中偏好的强度。实验结果证实,具有不同偏好强度的数据对奖励模型性能有不同的影响。我们引入了一系列新颖的方法来减轻数据集中不正确和模糊偏好的影响,并充分利用高质量的偏好数据。(2)从算法角度,我们引入对比学习来增强奖励模型区分被选择回复和被拒绝回复的能力,从而提高模型泛化性。此外,我们采用元学习使奖励模型能够在分布外样本中保持区分细微差异的能力,并且该方法可用于迭代式RLHF优化。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑