MulFeRL：在多轮循环中利用语言反馈增强强化学习
MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop
AI总结：
针对强化学习中标量奖励稀疏且缺乏信息的问题，提出MulFeRL框架，通过多轮语言反馈引导失败样本的再生、进度信用分配和结构化反馈注入，提升模型推理性能。
摘要：
具有可验证奖励的强化学习（RLVR）被广泛用于提升各领域的推理能力，但仅基于结果的标量奖励往往稀疏且信息量不足。这一限制对失败样本尤为严重，因为标量奖励仅指示解决方案不正确，而未解释推理为何失败。在本文中，我们利用更丰富的语言反馈来引导失败样本上的RLVR，并将反馈引发的进展转化为可训练的学习信号。我们提出MulFeRL（多轮反馈引导的强化学习），这是一个多轮、事件触发的RLVR框架，结合了用于反馈引导失败样本再生的进展诱导、用于从验证器确认的进展中学习的进展信用分配，以及用于将反馈整合到模型推理过程中的结构化反馈注入。在采样的OpenR1-Math上训练后，MulFeRL在领域内优于监督、自蒸馏和RLVR基线，同时展现出强大的领域外泛化能力。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning across domains, but outcome-only scalar rewards are often sparse and uninformative. This limitation is especially severe for failed samples, where scalar rewards indicate only that a solution is incorrect without explaining why the reasoning breaks down. In this paper, we leverage richer verbal feedback to guide RLVR on failed samples and convert feedback-induced progress into trainable learning signals. We propose MulFeRL (Multi-turn Feedback-guided Reinforcement Learning), a multi-turn, event-triggered RLVR framework that combines progress induction for feedback-guided regeneration of failed samples, progress credit assignment for learning from verifier-confirmed progress, and structured feedback injection for integrating feedback into the model's reasoning process. Trained on sampled OpenR1-Math, MulFeRL outperforms supervised, self-distillation-based, and RLVR baselines in-domain, while also showing strong out-of-domain generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
