ThinkTuning：无需蒸馏即可注入认知反思
ThinkTuning: Instilling Cognitive Reflections without Distillation
AI总结：
ThinkTuning提出一种基于GRPO的交互式训练方法，通过同等规模教师模型的纠正性反馈来引导不具备思考行为的学生模型发展自反思与多步推理能力，在多个基准上优于零样本和vanilla-GRPO基线。
摘要：
测试时扩展的最新进展催生了具备自反思行为和多步推理能力的思考型大语言模型（LLM）。虽然强化学习（RL）驱动了这一自我改进范式，但最近的一项研究（Gandhi et al., 2025）表明，仅靠RL并不能真正注入这些新的推理能力——它只是激发出基础模型中已经存在的行为。这引出了一个问题：对于那些本身不表现出此类思考行为的模型，我们究竟该如何训练它们，使其首先发展出这种能力？为此，我们提出了ThinkTuning，这是一种基于GRPO的交互式训练方法，在该方法中，我们用教师模型的指导来增强学生模型的 rollout。我们的方法受到课堂教学实践的简单启发：教师提出一个问题，让学生尝试作答，然后给出纠正性反馈——足以将思维引向正确方向，随后展示解决方案。每一段反馈都会重塑学生的思路，引导他们得出正确的解决方案。类似地，我们发现，通过来自同等规模教师模型的反馈所实现的这种隐式监督，能够提升学生模型的推理能力。具体而言，在多个基准测试上，我们的方法平均比零样本基线提高了3.85%；在MATH-500、AIME和GPQA-Diamond上，相比 vanilla-GRPO 基线分别提高了2.08%、2.23%和3.99%。源代码可在 https://github.com/3rdAT/ThinkTuning 获取。
英文摘要：
Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning. While RL drives this self-improvement paradigm, a recent study (Gandhi et al., 2025) shows that RL alone does not truly instill these new reasoning abilities - it merely draws out behaviors already present in the base models. This raises a question: How can we train the models that don't exhibit such thinking behavior to develop it in the first place? To this end, we propose ThinkTuning, a GRPO-based interactive training approach where we augment the rollouts of a student model with the guidance from a teacher model. A simple idea from classroom practice inspires our method: a teacher poses a problem, lets the student try an answer, then gives corrective feedback -- enough to point the mind in the right direction and then show the solution. Each piece of feedback reshapes the student's thoughts, leading them to arrive at the correct solution. Similarly, we find that this type of implicit supervision through feedback from a teacher model of the same size improves the reasoning capabilities of the student model. In particular, on average, our method shows a 3.85% improvement over zero-shot baselines across benchmarks, and on MATH-500, AIME and GPQA-Diamond it shows 2.08%, 2.23% and 3.99% improvements over the vanilla-GRPO baseline. Source code is available at https://github.com/3rdAT/ThinkTuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
