关于激发和改进类R1推理模型的实证研究
An Empirical Study on Eliciting and Improving R1-like Reasoning Models
AI总结：
本文系统研究了影响类R1推理模型强化学习训练的各种因素，证明该方法能有效提升基础和微调模型的推理能力，并发现引入工具操作可显著增强模型性能。
摘要：
在本报告中，我们呈现了作为STILL项目一部分的慢思考模型开发的第三份技术报告。随着技术路径日益清晰，扩展RL训练已成为实现此类推理模型的核心技术。我们系统地实验并记录了影响RL训练的各种因素的效果，在base models和fine-tuned models上均进行了实验。具体而言，我们证明我们的RL训练方法持续改进了Qwen2.5-32B base models，提升了响应长度和测试准确率。此外，我们表明，即使像DeepSeek-R1-Distill-Qwen-1.5B这样的模型已经达到了高性能水平，仍可通过RL训练进一步优化，在AIME 2024上达到39.33%的准确率。除RL训练外，我们还探索了工具操作的使用，发现它能显著提升大型推理模型的推理性能。该方法在AIME 2024上结合greedy search实现了86.67%的惊人准确率，凸显了其在增强模型能力方面的有效性。我们在STILL项目网站发布了相关资源：https://github.com/RUCAIBox/Slow_Thinking_with_LLMs。
英文摘要：
In this report, we present the third technical report on the development of slow-thinking models as part of the STILL project. As the technical pathway becomes clearer, scaling RL training has become a central technique for implementing such reasoning models. We systematically experiment with and document the effects of various factors influencing RL training, conducting experiments on both base models and fine-tuned models. Specifically, we demonstrate that our RL training approach consistently improves the Qwen2.5-32B base models, enhancing both response length and test accuracy. Furthermore, we show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already achieved a high performance level, it can be further refined through RL training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we also explore the use of tool manipulation, finding that it significantly boosts the reasoning performance of large reasoning models. This approach achieves a remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its effectiveness in enhancing model capabilities. We release our resources at the STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
