arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.24709cs.LGcs.CL
收藏

通过受限数据合成和渐进奖励训练LLM进行多步骤工具协调

Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards

Amazon Services Inc.(亚马逊服务公司);Department of Computer Science and Engineering, HKUST(香港科技大学计算机科学与工程系);School of Computational Science and Engineering, Georgia Institute of Technology(佐治亚理工学院计算科学与工程学院)

Cheng Jiayang, Xin Liu, Zhihan Zhang, Haoyang Wen, Zixuan Zhang, Qingyu Yin, Shiyang Li, Priyanka Nigam, Bing Yin, Chao Zhang, Yangqiu Song

AI总结:

本文提出强化学习框架解决LLM多步骤工具协调难题,通过构建确定性环境和渐进奖励机制提升执行准确率,并在不同API生态中实现技能迁移。

摘要:

多步骤工具协调仍对LLM构成挑战,因最先进模型常因参数错误在完整序列执行中失败。训练这些工作流面临两大障碍:缺乏支持复杂现实API依赖的环境,以及稀疏二元奖励无法提供部分正确性的信号。我们提出一个强化学习框架解决这两个挑战。首先,我们构建了一个由大规模真实API响应缓存支持的确定性环境,能够生成受控复杂度的有效多步骤轨迹。其次,我们引入了渐进奖励,将正确性分解为原子有效性(逐步细化的调用级正确性)和协调一致性(正确序列化并尊重依赖性)。在ComplexFuncBench上,我们的方法显著提高了回合准确率,消融实验确认了两种奖励组件的必要性。跨基准评估显示,在BFCL v4上,学习的协调技能转移到完全不同的API生态系统(如代理网络搜索和内存管理),产生一致的收益同时保持单步性能稳定。代码可在https://github.com/horizon-rl/ToolOrchestrationReward获取。

英文摘要:

Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑