RefCritic：利用细化反馈训练长链思维批评模型
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
AI总结：
本文提出RefCritic，一种基于强化学习和双重规则奖励的长链思维批评模型，通过解判断正确性与细化准确率生成可操作反馈，在多个基准上显著提升策略模型的批评与细化能力。
摘要：
随着大语言模型（LLMs）的快速发展，开发有效的批评模块以实现精确引导变得至关重要但也极具挑战。在本文中，我们首先证明，当前解决方案中广泛采用的通过监督微调构建批评模块的方法未能真正增强模型的批评能力，其产生的批评流于表面，缺乏充分的反思与验证。为了释放前所未有的批评能力，我们提出了RefCritic，这是一种基于强化学习的长链思维批评模块，采用双重基于规则的奖励：（1）实例层面的解判断正确性，以及（2）基于批评的策略模型细化准确率，旨在生成高质量评估和可操作的反馈，从而有效引导模型细化。我们在Qwen2.5-14B-Instruct和DeepSeek-R1-Distill-Qwen-14B上，跨五个基准对RefCritic进行了评估。在批评和细化设置下，RefCritic在所有基准上均表现出一致的优势，例如，在AIME25上，相应的基础模型分别获得了6.8%和7.2%的提升。值得注意的是，在多数投票下，经RefCritic筛选的策略模型随着投票数的增加展现出更优的扩展性。此外，尽管仅在解级别的监督上训练，RefCritic在ProcessBench（一个用于识别数学推理中错误步骤的基准）上仍优于步骤级监督方法。
英文摘要：
With the rapid advancement of Large Language Models (LLMs), developing effective critic modules for precise guidance has become crucial yet challenging. In this paper, we initially demonstrate that supervised fine-tuning for building critic modules (which is widely adopted in current solutions) fails to genuinely enhance models' critique abilities, producing superficial critiques with insufficient reflections and verifications. To unlock the unprecedented critique capabilities, we propose RefCritic, a long-chain-of-thought critic module based on reinforcement learning with dual rule-based rewards: (1) instance-level correctness of solution judgments and (2) refinement accuracies of the policy model based on critiques, aiming to generate high-quality evaluations with actionable feedback that effectively guides model refinement. We evaluate RefCritic on Qwen2.5-14B-Instruct and DeepSeek-R1-Distill-Qwen-14B across five benchmarks. On critique and refinement settings, RefCritic demonstrates consistent advantages across all benchmarks, e.g., 6.8\% and 7.2\% gains on AIME25 for the respective base models. Notably, under majority voting, policy models filtered by RefCritic show superior scaling with increased voting numbers. Moreover, despite training on solution-level supervision, RefCritic outperforms step-level supervised approaches on ProcessBench, a benchmark to identify erroneous steps in mathematical reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
