ReasonFlux：通过扩展思维模板实现分层大语言模型推理
ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates
AI总结：
ReasonFlux通过约500个高层思维模板库、模板轨迹上的分层强化学习及推理时自适应模板扩展，仅用8块GPU训练32B模型，在MATH和AIME基准上超越o1-preview与DeepSeek-V3，实现最先进的数学推理能力。
摘要：
我们提出，通过扩展思维模板实现的分层大语言模型推理能够有效优化推理搜索空间，并在数学推理能力上超越OpenAI o1-preview和DeepSeek V3等强大模型。我们仅使用8块GPU训练了ReasonFlux-32B模型，并引入三项创新：（i）一个结构化且通用的思维模板库，包含约500个高层思维模板，能够泛化到相似或相关的推理问题；（ii）在思维模板序列而非长思维链上执行分层强化学习，优化基础大语言模型以规划出最优模板轨迹，逐步处理复杂问题；（iii）一个全新的推理扩展系统，通过在推理时自适应扩展思维模板来实现分层大语言模型推理。凭借比DeepSeek-R1和o3-mini更具可解释性的模板轨迹推理结构，我们的ReasonFlux-32B将数学推理能力显著提升至最先进水平。值得注意的是，在MATH基准上，它达到了91.2%的准确率，超越o1-preview 6.7%。在美国数学奥林匹克（AIME）基准上，ReasonFlux-32B平均解决56.7%的问题，分别超越o1-preview和DeepSeek-V3 27%和45%。代码：https://github.com/Gen-Verse/ReasonFlux
英文摘要：
We present that hierarchical LLM reasoning via scaling thought templates can effectively optimize the reasoning search space and outperform the mathematical reasoning capabilities of powerful LLMs like OpenAI o1-preview and DeepSeek V3. We train our ReasonFlux-32B model with only 8 GPUs and introduces three innovations: (i) a structured and generic thought template library, containing around 500 high-level thought templates capable of generalizing to similar or relevant reasoning problems; (ii) performing hierarchical reinforcement learning on a sequence of thought templates instead of long CoTs, optimizing a base LLM to plan out an optimal template trajectory for gradually handling complex problems; (iii) a brand new inference scaling system that enables hierarchical LLM reasoning by adaptively scaling thought templates at inference time. With a template trajectory containing more explainable reasoning structures than DeepSeek-R1 and o3-mini, our ReasonFlux-32B significantly advances math reasoning capabilities to state-of-the-art levels. Notably, on the MATH benchmark, it achieves an accuracy of 91.2% and surpasses o1-preview by 6.7%. On the USA Math Olympiad (AIME) benchmark, ReasonFlux-32B solves an average of 56.7% of problems, surpassing o1-preview and DeepSeek-V3 by 27% and 45%, respectively. Code: https://github.com/Gen-Verse/ReasonFlux
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
