arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10216cs.CLcs.AI
收藏

正则化隐藏状态使 LLM 能学习可泛化的奖励模型

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Georgia Institute of Technology(佐治亚理工学院);Princeton University(普林斯顿大学)

Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, Tong Zhang

AI总结:

本文提出通过保留语言模型头并引入文本生成损失来正则化隐藏状态,从而提升奖励模型在分布外任务上的泛化能力并缓解 RLHF 中的奖励过度优化问题。

摘要:

在基于人类反馈的强化学习(RLHF)框架中,基于人类偏好数据训练的奖励模型已被证明能有效使大语言模型(LLMs)与人类意图对齐。然而,当前奖励模型对未见过的提示和响应的泛化能力有限,这可能导致一种称为奖励过度优化的意外现象,即由于对奖励的过度优化而导致实际性能下降。以往研究主张约束策略优化,而本研究引入了一种新方法,通过正则化隐藏状态来增强奖励模型对分布偏移的泛化能力。具体而言,我们保留基础模型的语言模型头,并引入一组文本生成损失以保持隐藏状态的文本生成能力,同时在相同的隐藏状态之后学习一个奖励头。我们的实验结果表明,所引入的正则化技术显著提高了所学奖励模型在各种分布外(OOD)任务上的准确性,并有效缓解了 RLHF 中的过度优化问题,提供了一种更可靠且更稳健的偏好学习范式。

英文摘要:

Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in a decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy optimization, our study introduces a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capabilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety of out-of-distribution (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering a more reliable and robust preference learning paradigm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑