StepTool：通过步粒度强化学习增强大语言模型的多步工具使用能力
StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
AI总结：
本文提出StepTool步粒度强化学习框架，通过步粒度奖励塑造和步粒度优化将工具学习建模为动态决策过程，在多个基准上显著提升LLMs多步工具使用的任务通过率和工具召回率。
摘要：
尽管大语言模型（LLMs）具备强大的文本生成能力，但在有效利用外部工具解决复杂任务方面仍面临困难，这一挑战被称为工具学习。现有方法主要依赖监督微调（supervised fine-tuning），将工具学习视为文本生成问题，而忽视了多步上下文中固有的决策复杂性。在本工作中，我们提出将工具学习建模为动态决策过程，并引入StepTool，这是一个新颖的步粒度强化学习框架，用于增强LLMs在多步工具使用中的能力。StepTool包含两个关键组件：步粒度奖励塑造（Step-grained Reward Shaping），根据每次工具调用的成功与否及其对任务完成的贡献为每次工具交互分配奖励；以及步粒度优化（Step-grained Optimization），应用策略梯度方法在多个决策步骤上优化模型。在多个基准上的大量实验表明，StepTool在任务通过率（Pass Rate）和相关工具召回率（Recall）方面始终优于基于SFT和基于RL的基线方法。此外，我们的分析表明，StepTool帮助模型发现新的工具使用策略，而不仅仅是对已有知识进行重新加权。这些结果凸显了细粒度决策建模在工具学习中的重要性，并确立了StepTool作为增强LLMs多步工具使用能力的通用且稳健的解决方案。代码和数据可在https://github.com/yuyq18/StepTool获取。
英文摘要：
Despite their powerful text generation capabilities, large language models (LLMs) still struggle to effectively utilize external tools to solve complex tasks, a challenge known as tool learning. Existing methods primarily rely on supervised fine-tuning, treating tool learning as a text generation problem while overlooking the decision-making complexities inherent in multi-step contexts. In this work, we propose modeling tool learning as a dynamic decision-making process and introduce StepTool, a novel step-grained reinforcement learning framework that enhances LLMs' capabilities in multi-step tool use. StepTool comprises two key components: Step-grained Reward Shaping, which assigns rewards to each tool interaction based on its invocation success and contribution to task completion; and Step-grained Optimization, which applies policy gradient methods to optimize the model across multiple decision steps. Extensive experiments across diverse benchmarks show that StepTool consistently outperforms both SFT-based and RL-based baselines in terms of task Pass Rate and Recall of relevant tools. Furthermore, our analysis suggests that StepTool helps models discover new tool-use strategies rather than merely re-weighting prior knowledge. These results highlight the importance of fine-grained decision modeling in tool learning and establish StepTool as a general and robust solution for enhancing multi-step tool use in LLMs. Code and data are available at https://github.com/yuyq18/StepTool.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
