DeepCritic：基于大型语言模型的深思熟虑式评论
DeepCritic: Deliberate Critique with Large Language Models
AI总结：
提出两阶段框架DeepCritic，通过监督微调与强化学习开发具备逐步深思评论能力的LLM评论模型，显著提升数学推理错误识别与生成器纠错效果。
摘要：
随着大型语言模型（LLMs）的快速发展，对其输出提供准确反馈和可扩展的监督成为一个紧迫且关键的问题。利用LLMs作为评论模型以实现自动化监督是一种有前景的解决方案。在本研究中，我们专注于研究和增强LLMs的数学评论能力。当前的LLM评论者对每个步骤的评论过于浅显和表面化，导致判断准确率低，且难以向LLM生成器提供充足反馈以纠正错误。为解决此问题，我们提出了一种新颖且有效的两阶段框架，以开发能够对数学解法的每个推理步骤进行深思熟虑式评论的LLM评论者。在第一阶段，我们利用Qwen2.5-72B-Instruct生成4.5K条长篇评论作为监督微调的种子数据。每条种子评论包含深思熟虑的逐步评论，涵盖多视角验证以及对每个推理步骤初始评论的深入剖析。随后，我们在微调模型上进行强化学习，使用来自PRM800K的人工标注数据或通过基于蒙特卡洛采样的正确性估计自动标注的数据，以进一步激发其评论能力。我们基于Qwen2.5-7B-Instruct开发的评论模型不仅在多个错误识别基准上显著优于现有LLM评论者（包括同等规模的DeepSeek-R1-distill模型和GPT-4o），还能通过更详细的反馈更有效地帮助LLM生成器修正错误步骤。
英文摘要：
As Large Language Models (LLMs) are rapidly evolving, providing accurate feedback and scalable oversight on their outputs becomes an urgent and critical problem. Leveraging LLMs as critique models to achieve automated supervision is a promising solution. In this work, we focus on studying and enhancing the math critique ability of LLMs. Current LLM critics provide critiques that are too shallow and superficial on each step, leading to low judgment accuracy and struggling to offer sufficient feedback for the LLM generator to correct mistakes. To tackle this issue, we propose a novel and effective two-stage framework to develop LLM critics that are capable of deliberately critiquing on each reasoning step of math solutions. In the first stage, we utilize Qwen2.5-72B-Instruct to generate 4.5K long-form critiques as seed data for supervised fine-tuning. Each seed critique consists of deliberate step-wise critiques that includes multi-perspective verifications as well as in-depth critiques of initial critiques for each reasoning step. Then, we perform reinforcement learning on the fine-tuned model with either existing human-labeled data from PRM800K or our automatically annotated data obtained via Monte Carlo sampling-based correctness estimation, to further incentivize its critique ability. Our developed critique model built on Qwen2.5-7B-Instruct not only significantly outperforms existing LLM critics (including the same-sized DeepSeek-R1-distill models and GPT-4o) on various error identification benchmarks, but also more effectively helps the LLM generator refine erroneous steps through more detailed feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
