arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.06326cs.CL
收藏

教授思考模型进行工具推理:一种完整的全管道配方用于工具集成推理

Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning

Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学)

Qianjia Cheng, Yuchen Zhang, Zhilin Wang, Yuxin Zuo, Shunkai Zhang, Yuchen Fan, Yu Qiao, Bowen Zhou, Ning Ding, Yu Cheng, Yun Luo, Ganqu Cui

AI总结:

本文提出一种全管道配方,通过注入自然工具使用行为提升强思考模型的工具集成推理能力,同时保持文本推理能力,展示在Qwen3模型上取得最佳性能。

摘要:

工具集成推理(TIR)提供了一种直接扩展思考模型超越纯文本推理限制的方法。然而,我们发现即使强思考模型几乎不调用工具,工具启用评估仍可能降低推理性能。本文研究如何在不牺牲无工具推理能力的情况下,将自然工具使用行为注入强思考模型,并提出全面的TIR配方。我们指出(i)TIR监督微调(SFT)的有效性取决于教师轨迹的可学习性,应优先选择适合工具增强解决方案的问题;(ii)控制工具使用轨迹的比例可缓解纯文本推理能力的灾难性遗忘;(iii)优化pass@k和响应长度而非训练损失可最大化TIR SFT收益并保留强化学习(RL)探索的余地;(iv)基于合适SFT初始化和明确防止模式崩溃的稳定RL与可验证奖励(RLVR)阶段提供简单而高效的解决方案。当应用于4B和30B规模的Qwen3思考模型时,我们的配方产生在多个开源模型基准测试中取得最佳性能的模型,例如在AIME 2025上分别达到96.7%和99.2%。

英文摘要:

Tool-integrated reasoning (TIR) offers a direct way to extend thinking models beyond the limits of text-only reasoning. Paradoxically, we observe that tool-enabled evaluation can degrade reasoning performance even when the strong thinking models make almost no actual tool calls. In this paper, we investigate how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability, and present a comprehensive TIR recipe. We highlight that (i) the effectiveness of TIR supervised fine-tuning (SFT) hinges on the learnability of teacher trajectories, which should prioritize problems inherently suited for tool-augmented solutions; (ii) controlling the proportion of tool-use trajectories could mitigate the catastrophic forgetting of text-only reasoning capacity; (iii) optimizing for pass@k and response length instead of training loss could maximize TIR SFT gains while preserving headroom for reinforcement learning (RL) exploration; (iv) a stable RL with verifiable rewards (RLVR) stage, built upon suitable SFT initialization and explicit safeguards against mode collapse, provides a simple yet remarkably effective solution. When applied to Qwen3 thinking models at 4B and 30B scales, our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models, such as 96.7% and 99.2% on AIME 2025 for 4B and 30B, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑