arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.00782cs.LG
收藏

基于人类反馈的强化学习中免费获得的密集奖励

Dense Reward for Free in Reinforcement Learning from Human Feedback

University of Cambridge(剑桥大学)

Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar

AI总结:

针对RLHF中LLM仅获回合末稀疏奖励难以优化的问题,提出利用奖励模型的注意力权重免费重分配奖励以密集化信号,理论上等价于势函数奖励塑形且不改变最优策略,实验证实可稳定训练、加速学习并可能得到更优局部最优解。

摘要:

基于人类反馈的强化学习(Reinforcement Learning from Human Feedback, RLHF)被认为是使大语言模型(Large Language Models, LLMs)能够有效遵循指令并提供有用辅助的关键进展。经典的RLHF流程中,LLM会针对查询生成补全内容,随后使用独立的奖励模型为整个补全内容分配分数。作为一个自回归过程,LLM需要采取多次“动作”(选择单个token),却仅在回合结束时获得单一的稀疏奖励,这种设置在传统强化学习中被认为难以优化。本研究利用了奖励模型包含的信息远超其标量输出这一特性——具体而言,奖励模型作为Transformer架构的一部分,会计算token上的注意力图。我们利用这些注意力权重在整个补全内容上重新分配奖励,在不产生额外计算成本、也无需任何额外建模的前提下,有效将奖励信号密集化并突出最重要的token。我们从理论上证明,该方法等价于基于势函数的奖励塑形,可确保最优策略保持不变。实验结果表明,该方法能够稳定训练、加快学习速度,且在实际场景中可能得到更优的局部最优解。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the LLM has to take many "actions" (selecting individual tokens) and only receives a single, sparse reward at the end of an episode, a setup that is known to be difficult to optimise in traditional reinforcement learning. In this work we leverage the fact that the reward model contains more information than just its scalar output, in particular, it calculates an attention map over tokens as part of the transformer architecture. We use these attention weights to redistribute the reward along the whole completion, effectively densifying the signal and highlighting the most important tokens, all without incurring extra computational cost or requiring any additional modelling. We demonstrate that, theoretically, this approach is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑