rStar-Math：小型 LLM 可通过自我进化的深度思考掌握数学推理
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
AI总结：
提出 rStar-Math，通过 MCTS、代码增强 CoT 合成、PPM 训练与四轮自我进化，让小型 LLM 无需强模型蒸馏即可达到领先数学推理水平。
摘要：
我们提出 rStar-Math，以证明小型语言模型（SLM）能够匹敌甚至超越 OpenAI o1 的数学推理能力，而无需从更强大的模型进行蒸馏。rStar-Math 通过 Monte Carlo Tree Search（MCTS，蒙特卡洛树搜索）运用“深度思考”来实现这一点，其中数学策略 SLM 在基于 SLM 的过程奖励模型指导下执行测试时搜索。rStar-Math 引入三项创新以解决训练这两个 SLM 的挑战：（1）一种新颖的代码增强 CoT 数据合成方法，通过大量 MCTS rollout 生成逐步验证的推理轨迹，用于训练策略 SLM；（2）一种新颖的过程奖励模型训练方法，避免朴素的步骤级分数标注，从而得到更有效的过程偏好模型（PPM）；（3）一套自我进化方案，其中策略 SLM 和 PPM 从零开始构建，并迭代进化以提升推理能力。通过 4 轮自我进化，并为 747k 道数学题合成数百万个解答，rStar-Math 将 SLM 的数学推理提升到最先进水平。在 MATH 基准上，它将 Qwen2.5-Math-7B 从 58.8% 提升到 90.0%，将 Phi3-mini-3.8B 从 41.4% 提升到 86.4%，分别比 o1-preview 高出 +4.5% 和 +0.9%。在 USA Math Olympiad（AIME）上，rStar-Math 平均解出 53.3%（8/15）的问题，跻身最聪明高中数学学生的前 20%。代码和数据将在 https://github.com/microsoft/rStar 提供。
英文摘要：
We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising "deep thinking" through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data sythesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids naïve step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs' math reasoning to state-of-the-art levels. On the MATH benchmark, it improves Qwen2.5-Math-7B from 58.8% to 90.0% and Phi3-mini-3.8B from 41.4% to 86.4%, surpassing o1-preview by +4.5% and +0.9%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53.3% (8/15) of problems, ranking among the top 20% the brightest high school math students. Code and data will be available at https://github.com/microsoft/rStar.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
