arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.20342cs.CLcs.AI
收藏

教语言模型使用工具进行推理

Teaching Language Models to Reason with Tools

University of Science and Technology of China(中国科学技术大学);Alibaba Inc.(阿里巴巴公司);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen International Center for Industrial and Applied Mathematics(深圳国际工业与应用数学中心);Shenzhen Research Institute of Big Data(深圳大数据研究院)

Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, Dayiheng Liu

AI总结:

提出CoRT框架,通过提示工程和强化学习训练语言模型有效使用代码解释器,在数学推理上提升准确率并大幅减少令牌使用。

摘要:

大型推理模型(LRMs),如OpenAI-o1,在自然语言推理方面展现出了令人印象深刻的能力。然而,这些模型在处理复杂的数学运算时,经常表现出低效或不准确。虽然集成诸如代码解释器(CIs)之类的计算工具提供了一种有前景的解决方案,但它引入了一个关键挑战:模型内部的概率推理与CI提供的外部确定性知识之间的冲突,这常常导致模型陷入无益的深思。为了克服这一点,我们引入了CoRT(代码优化推理训练),一个旨在教LRMs有效利用CIs的后训练框架。我们提出了提示工程(Hint-Engineering),一种新的数据合成策略,该策略在推理路径中的最优位置策略性地注入各种提示。这种方法生成了高质量、代码集成的推理数据,专门用于优化LRM-CI交互。使用这种方法,我们合成了30个高质量样本,通过监督微调对参数规模从1.5B到32B的模型进行后训练。CoRT通过采用拒绝采样和强化学习,进一步优化了外部CI使用和内部思考的多轮交错。我们的实验评估证明了CoRT的有效性,在五个具有挑战性的数学推理数据集上,DeepSeek-R1-Distill-Qwen-32B和DeepSeek-R1-Distill-Qwen-1.5B分别取得了4%和8%的绝对改进。此外,CoRT显著提高了效率,与纯自然语言推理基线相比,32B模型和1.5B模型的令牌使用量分别减少了约30%和50%。模型和代码可在https://github.com/ChengpengLi1003/CoRT获取。

英文摘要:

Large reasoning models (LRMs) like OpenAI-o1 have shown impressive capabilities in natural language reasoning. However, these models frequently demonstrate inefficiencies or inaccuracies when tackling complex mathematical operations. While integrating computational tools such as Code Interpreters (CIs) offers a promising solution, it introduces a critical challenge: a conflict between the model's internal, probabilistic reasoning and the external, deterministic knowledge provided by the CI, which often leads models to unproductive deliberation. To overcome this, we introduce CoRT (Code-Optimized Reasoning Training), a post-training framework designed to teach LRMs to effectively utilize CIs. We propose \emph{Hint-Engineering}, a new data synthesis strategy that strategically injects diverse hints at optimal points within reasoning paths. This approach generates high-quality, code-integrated reasoning data specifically tailored to optimize LRM-CI interaction. Using this method, we have synthesized 30 high-quality samples to post-train models ranging from 1.5B to 32B parameters through supervised fine-tuning. CoRT further refines the multi-round interleaving of external CI usage and internal thinking by employing rejection sampling and reinforcement learning. Our experimental evaluations demonstrate CoRT's effectiveness, yielding absolute improvements of 4\% and 8\% on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B, respectively, across five challenging mathematical reasoning datasets. Moreover, CoRT significantly enhances efficiency, reducing token usage by approximately 30\% for the 32B model and 50\% for the 1.5B model compared to pure natural language reasoning baselines. The models and code are available at: https://github.com/ChengpengLi1003/CoRT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑