arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.19409cs.CL
收藏

利用演示引导强化学习抑制大语言模型中的奖励过度优化

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning

INRIA(法国国家信息与自动化研究所);Cohere(Cohere公司);DeepMind(DeepMind公司);EHESS(法国高等社会科学研究院);ENS-PSL(巴黎高等师范学院-巴黎文理研究大学);CNRS(法国国家科学研究中心);Meta AI Research(Meta人工智能研究院)

Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, Olivier Pietquin

AI总结:

本文提出基于演示的奖励校准方法RCfD,通过最小化演示与LLM奖励之间的距离来替代直接最大化奖励,从而在无需复杂KL超参数调优的情况下抑制奖励过度优化,并在三个语言任务上验证了其有效性。

摘要:

虽然强化学习已被证明对于微调大语言模型至关重要,但它可能导致奖励过度优化。现有方法通过添加KL正则化来解决奖励过度优化问题,这需要计算成本高昂的超参数调优。此外,KL正则化仅专注于对语言策略进行正则化,忽略了正则化的一个潜在来源:奖励函数本身。受演示引导强化学习的启发,我们在此提出基于演示的奖励校准(RCfD),它利用人类演示和奖励模型来重新校准奖励目标。形式上,给定一个提示,RCfD目标是最小化演示奖励与大语言模型奖励之间的距离,而不是直接最大化奖励函数。这种目标转变避免激励大语言模型利用奖励模型,并促进更自然和多样化的语言生成。我们在三个语言任务上展示了RCfD的有效性,它在缓解奖励过度优化的同时,取得了与精心调优的基线相当的性能。

英文摘要:

While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑