arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.08274cs.CL
收藏

发现与强化工具集成推理链 via 滚出树

Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees

The Chinese University of Hong Kong(香港中文大学);Tencent(腾讯);Tsinghua University(清华大学)

Kun Li, Zenan Xu, Junan Li, Zengrui Jin, Jinghao Deng, Zexuan Qiu, Bo Zhou

AI总结:

DART通过强化学习框架在长链推理中自发使用工具,无需人工标注,有效提升工具集成推理性能。

摘要:

工具集成推理已成为增强大语言模型(LLMs)的计算能力的关键范式,然而将工具使用整合到长链推理(长CoT)中仍处于探索阶段,主要由于训练数据稀缺以及在不损害模型内在长链推理能力的情况下整合工具使用具有挑战性。在本文中,我们引入DART(via 滚出树发现与强化工具集成推理链),一种强化学习框架,能够在长CoT推理中自发地使用工具,而无需人工标注。DART通过在训练过程中构建动态滚出树来发现有效的工具使用机会,在有希望的位置分支以探索多样的工具集成轨迹。随后,基于树的过程优势估计识别并奖励特定子轨迹中工具调用对解决方案的积极贡献,从而有效强化这些有益行为。在具有挑战性的基准测试如AIME和GPQA-Diamond上进行的广泛实验表明,DART显著优于现有方法,成功地将工具执行与长CoT推理协调一致。

英文摘要:

Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑