超越稀疏奖励：在文本生成中利用语言模型评价增强强化学习
Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
AI总结：
本文提出一种利用大型语言模型评价能力在强化学习训练中生成中间步骤奖励的新框架，通过将反馈转化为token或span级别奖励，显著提升了文本生成任务中策略模型的样本效率和整体性能。
摘要：
强化学习（RL）可以使语言模型与不可微的奖励信号（如人类偏好）对齐。然而，这些奖励信号的稀疏性带来了一个主要挑战——通常，整个输出只有一个奖励。这种奖励的稀疏性会导致学习效率低下且不稳定。为了应对这一挑战，本文引入了一种新颖的框架，利用大型语言模型（LLMs）的评价能力在RL训练期间产生中间步骤的奖励。我们的方法涉及将策略模型与评价语言模型耦合，后者负责对输出的每个部分提供全面的反馈。然后，这种反馈被转化为可用于指导RL训练过程的token或span级别的奖励。我们在两种不同的设置下研究了这种方法：一种是策略模型较小并与更强大的评价模型配对，另一种是单个语言模型同时承担这两个角色。我们在三个文本生成任务上评估了我们的方法：情感控制、语言模型去毒和摘要生成。实验结果表明，结合人工内在奖励显著提高了样本效率和策略模型的整体性能，这得到了自动评估和人类评估的支持。
英文摘要：
Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of these reward signals - typically, there is only a single reward for an entire output. This sparsity of rewards can lead to inefficient and unstable learning. To address this challenge, our paper introduces an novel framework that utilizes the critique capability of Large Language Models (LLMs) to produce intermediate-step rewards during RL training. Our method involves coupling a policy model with a critic language model, which is responsible for providing comprehensive feedback of each part of the output. This feedback is then translated into token or span-level rewards that can be used to guide the RL training process. We investigate this approach under two different settings: one where the policy model is smaller and is paired with a more powerful critic model, and another where a single language model fulfills both roles. We assess our approach on three text generation tasks: sentiment control, language model detoxification, and summarization. Experimental results show that incorporating artificial intrinsic rewards significantly improve both sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
