SuperCorrect：利用思维模板蒸馏与自我纠正提升小型LLM推理能力
SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction
AI总结：
提出SuperCorrect两阶段框架，利用教师模型思维模板蒸馏和跨模型DPO自我纠正，显著提升小型LLM数学推理能力，在7B模型中取得SOTA性能。
摘要：
大型语言模型（LLMs）如GPT-4、DeepSeek-R1和ReasonFlux在各种推理任务中展现出显著进步。然而，较小型的LLM在复杂数学推理方面仍然表现不佳，因为它们难以有效识别和纠正推理错误。近期基于反思的方法旨在通过实现自我反思和自我纠正来解决这些问题，但它们在独立检测推理步骤中的错误方面仍面临挑战。为克服这些局限，我们提出SuperCorrect，一种新颖的两阶段框架，利用大型教师模型来监督和纠正较小学生模型的推理与反思过程。在第一阶段，我们从教师模型中提取分层的高层和详细思维模板，以引导学生模型产生更细粒度的推理思维。在第二阶段，我们引入跨模型协作的直接偏好优化（DPO），通过在训练过程中遵循教师的纠正轨迹来增强学生模型的自我纠正能力。这种跨模型DPO方法教会学生模型有效定位并解决错误思维，借助教师模型的错误驱动洞察，突破其思维瓶颈，获取新技能和知识以应对挑战性问题。大量实验一致表明我们优于先前方法。值得注意的是，我们的SuperCorrect-7B模型在MATH/GSM8K基准上显著超越强大的DeepSeekMath-7B达7.8%/5.3%，超越Qwen2.5-Math-7B达15.1%/6.3%，在所有7B模型中实现新的最先进（SOTA）性能。代码：https://github.com/YangLing0818/SuperCorrect-llm
英文摘要：
Large language models (LLMs) like GPT-4, DeepSeek-R1, and ReasonFlux have shown significant improvements in various reasoning tasks. However, smaller LLMs still struggle with complex mathematical reasoning because they fail to effectively identify and correct reasoning errors. Recent reflection-based methods aim to address these issues by enabling self-reflection and self-correction, but they still face challenges in independently detecting errors in their reasoning steps. To overcome these limitations, we propose SuperCorrect, a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model. In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts. In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model by following the teacher's correction traces during training. This cross-model DPO approach teaches the student model to effectively locate and resolve erroneous thoughts with error-driven insights from the teacher model, breaking the bottleneck of its thoughts and acquiring new skills and knowledge to tackle challenging problems. Extensive experiments consistently demonstrate our superiority over previous methods. Notably, our SuperCorrect-7B model significantly surpasses powerful DeepSeekMath-7B by 7.8%/5.3% and Qwen2.5-Math-7B by 15.1%/6.3% on MATH/GSM8K benchmarks, achieving new SOTA performance among all 7B models. Code: https://github.com/YangLing0818/SuperCorrect-llm
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
