模仿、探索与自我改进：关于慢思考推理系统的复现报告
Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
AI总结：
本文提出 STILL-2 框架，通过模仿、探索与自我改进机制训练慢思考推理模型，在三个基准上取得与业界系统相当的竞争力。
摘要：
最近，诸如 o1 等慢思考推理系统在解决复杂推理任务中展现出了卓越的能力。这些系统通常在响应查询之前进行长时间的思考过程，使其能够生成更全面、准确且经过充分推理的解决方案。这些系统主要由业界开发和维护，其核心技术并未公开。作为回应，研究社区中越来越多的研究旨在探索这些强大推理系统背后的技术基础。在这些先前工作的基础上，本文提出了一份关于实现类 o1 推理系统的复现报告。我们引入了一个“模仿、探索与自我改进”的框架，记为 STILL-2，作为我们训练推理模型的主要技术方法。在初始阶段，我们使用蒸馏的长篇思考数据对推理模型进行微调，使其能够调用慢思考模式。随后，鼓励模型通过生成多个 rollout 来探索具有挑战性的问题，这可以产生越来越多能够导向正确答案的高质量轨迹。此外，模型通过迭代地完善其训练数据集来进行自我改进。为了验证该方法的有效性，我们在三个具有挑战性的基准上进行了广泛的实验。实验结果表明，与业界级别的推理系统相比，我们的方法在这些基准上取得了具有竞争力的性能。
英文摘要：
Recently, slow-thinking reasoning systems, such as o1, have demonstrated remarkable capabilities in solving complex reasoning tasks. These systems typically engage in an extended thinking process before responding to a query, allowing them to generate more thorough, accurate, and well-reasoned solutions. These systems are primarily developed and maintained by industry, with their core techniques not publicly disclosed. In response, an increasing number of studies from the research community aim to explore the technical foundations underlying these powerful reasoning systems. Building on these prior efforts, this paper presents a reproduction report on implementing o1-like reasoning systems. We introduce an ``imitate, explore, and self-improve'' framework, denoted as \textbf{STILL-2}, as our primary technical approach to train the reasoning model. In the initial phase, we use distilled long-form thought data to fine-tune the reasoning model, enabling it to invoke a slow-thinking mode. The model is then encouraged to explore challenging problems by generating multiple rollouts, which can result in increasingly more high-quality trajectories that lead to correct answers. Furthermore, the model undergoes self-improvement by iteratively refining its training dataset. To verify the effectiveness of this approach, we conduct extensive experiments on three challenging benchmarks. The experimental results demonstrate that our approach achieves competitive performance compared to industry-level reasoning systems on these benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
