通过文本反馈扩展强化学习的能力
Expanding the Capabilities of Reinforcement Learning via Text Feedback
AI总结：
本文提出通过文本反馈改进强化学习的方法，通过自我蒸馏和反馈建模在推理、数学和写作任务中超越基线性能。
摘要：
强化学习在大语言模型后训练中的成功源于一个不合理不 informative 的来源：每个 rollout 只提供一个信息位作为二进制奖励或偏好标签。在另一方面，蒸馏提供密集的监督，但需要演示，这些演示成本高且难以扩展。我们研究文本反馈作为中间信号：比标量奖励更丰富，但比完整演示更便宜。文本反馈是人类互动的自然模式，在许多现实世界场景中已经广泛存在，用户、标注员和自动化裁判经常批评 LLM 输出。为了在大规模上利用文本反馈，我们正式化了一种多轮强化学习设置，即文本反馈强化学习（RLTF），其中文本反馈在训练期间可用，但在推理时不可用。因此，模型必须学会内化反馈以提高测试时单轮性能。为此，我们提出两种方法：自我蒸馏（RLTF-SD），它训练单轮策略以匹配其自身反馈条件下的第二轮生成；以及反馈建模（RLTF-FM），它将反馈作为辅助目标进行预测。我们对两种方法进行了理论分析，并在推理谜题、竞赛数学和创意写作任务上进行了实证评估。我们的结果表明，两种方法在多个基准上均优于强大的基线，突显了在大规模上使用额外丰富监督的强化学习的潜力。
英文摘要：
The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study text feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, annotators, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
