arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.11971cs.LGcs.AI
收藏

通过组不变学习提升人类偏好对齐的泛化能力

Improving Generalization of Alignment with Human Preferences through Group Invariant Learning

Fudan University(复旦大学);ByteDance Inc.(字节跳动有限公司)

Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

针对强化学习在人类偏好对齐中易走捷径、泛化不足的问题,本文提出通过自动分组并最大化组间差异,优化困难组并自适应调整探索空间,从而提升训练稳定性与模型泛化能力。

摘要:

基于语言模型(LLM)的AI助手的成功关键依赖于从人类反馈中进行强化学习(RLHF),这使得模型能够生成更符合人类偏好的响应。作为通用AI助手,人们越来越期望它们在各个领域表现一致。然而,先前的研究表明,强化学习(RL)常常利用捷径来获得高奖励,而忽视具有挑战性的样本。这种对快速奖励获取的关注既损害了训练的稳定性,也削弱了模型对新的、未见数据的泛化能力。在这项工作中,我们提出了一种新颖的方法,能够通过强化学习在各种数据组或领域之间学习一致的策略。鉴于获取组标注存在挑战,我们的方法自动将数据分类为不同的组,并刻意最大化性能差异。然后,我们优化策略以在具有挑战性的组上表现良好。最后,利用已建立的组,我们的方法自适应地调整探索空间,将更多的学习能力分配给更具挑战性的数据,并防止模型在较简单的数据上过度优化。实验结果表明,我们的方法显著提升了训练稳定性和模型泛化能力。

英文摘要:

The success of AI assistants based on language models (LLMs) hinges crucially on Reinforcement Learning from Human Feedback (RLHF), which enables the generation of responses more aligned with human preferences. As universal AI assistants, there's a growing expectation for them to perform consistently across various domains. However, previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. This focus on quick reward gains undermines both the stability in training and the model's ability to generalize to new, unseen data. In this work, we propose a novel approach that can learn a consistent policy via RL across various data groups or domains. Given the challenges associated with acquiring group annotations, our method automatically classifies data into different groups, deliberately maximizing performance variance. Then, we optimize the policy to perform well on challenging groups. Lastly, leveraging the established groups, our approach adaptively adjusts the exploration space, allocating more learning capacity to more challenging data and preventing the model from over-optimizing on simpler data. Experimental results indicate that our approach significantly enhances training stability and model generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑