Double-Checker：通过自我批评微调提升慢思考大语言模型的推理能力
Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
AI总结：
针对慢思考大语言模型自我批评与方案优化能力不足的问题，提出Double-Checker框架，通过1730个实例微调实现迭代式自我批评优化，在AIME基准上将pass@1从4.4%提升至18.2%。
摘要：
尽管慢思考大语言模型（LLMs）展现出类似反思的推理能力（通常被称为“顿悟时刻”），但其生成有价值的批评并优化先前解决方案的能力仍然有限。本文提出了Double-Checker，这是一个系统性框架，旨在通过促进慢思考LLMs对其先前解决方案进行明确的自我批评和迭代优化，提升其推理能力。通过在我们精心整理的1730个自我批评实例上进行微调，Double-Checker使长思维链（CoT）LLMs能够在推理过程中迭代地批评和优化其输出，直到在自身生成的批评下判定解决方案正确为止。我们在一套全面的推理基准上验证了Double-Checker的有效性，证明迭代式自我批评显著提升了长CoT LLMs的推理能力。值得注意的是，与原始长CoT LLMs相比，我们的Double-Checker将具有挑战性的AIME基准上的pass@1性能从4.4%提升至18.2%。这些结果为开发更可靠、更有效且具备结构化自我批评能力的LLMs指明了一个有前景的方向。我们的代码和数据可在https://github.com/XinXU-USTC/DoubleChecker获取。
英文摘要：
While slow-thinking large language models (LLMs) exhibit reflection-like reasoning, commonly referred to as the "aha moment:, their ability to generate informative critiques and refine prior solutions remains limited. In this paper, we introduce Double-Checker, a principled framework designed to enhance the reasoning capabilities of slow-thinking LLMs by fostering explicit self-critique and iterative refinement of their previous solutions. By fine-tuning on our curated 1,730 self-critical instances, Double-Checker empowers long-CoT LLMs to iteratively critique and refine their outputs during inference until they evaluate their solutions as correct under self-generated critiques. We validate the efficacy of Double-Checker across a comprehensive suite of reasoning benchmarks, demonstrating that iterative self-critique significantly enhances the reasoning capabilities of long-CoT LLMs. Notably, our Double-Checker increases the pass@1 performance on challenging AIME benchmarks from 4.4% to 18.2% compared to the original long-CoT LLMs. These results highlight a promising direction for developing more trustworthy and effective LLMs capable of structured self-critique. Our codes and data are available at https://github.com/XinXU-USTC/DoubleChecker
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
