arXiv 2605.175582026-05-17cs.SEcs.CL

Firefly：从真实API生成大规模验证工具调用数据

Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs

Northeastern University（东北大学）；Independent Researcher（独立研究者）

Yuxuan Lu, Ziyi Wang, Yingzhou Lu, Yisi Sang, Jiri Gesi, Xianfeng Tang, Yimeng Zhang, Zhenwei Dai, Hui Liu, Hanqing Lu, Chen Luo, Qi He, Benoit Dumoulin, Jing Huang, Dakuo Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出FireFly框架，通过反向合成方法生成具有验证标签的工具调用数据，利用强LLM探索真实API并构建工具图，以提高大规模工具空间下的训练效率和结果准确性。

摘要：

训练工具调用代理需要大规模轨迹数据并具有可验证的标签，但现有方法要么合成与真实API行为不同的环境，要么生成无地面真实结果的任务。我们提出了FireFly，一个从真实世界MCP服务器生成验证工具调用数据的流水线。我们的关键见解是反转标准合成流程：而不是生成任务并希望它们可解决，我们首先让强大的LLM沿着图引导的DAG结构探索真实API，然后从观察到的结果反向合成任务，通过构造确保标签的正确性。为了处理真实世界工具空间（约1,000个工具）的规模，我们构建了一个成对的工具图并采样子DAGs，以专注于语义上连贯的工作流程。为了解决实时API中的环境漂移问题，我们构建了一个检索增强的模拟器，缓存所有探索结果并在训练和评估期间回放它们，使完全离线且可重复的RL成为可能。应用此流水线产生了5,144个经过验证的任务，覆盖240个服务器和993个工具。在FireFly上训练的40亿参数模型使用GRPO方法在我们的保留测试集上与Claude Sonnet 4.6相当，并在多个工具调用基准测试（包括Tau2-Bench、MCPMark和MCP-Atlas）上显示出改进。

英文摘要：

Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environments that diverge from real API behavior or generate tasks without ground-truth outcomes for verification. We present FireFly, a pipeline for generating verified tool-call data from real-world MCP servers. Our key insight is to invert the standard synthesis pipeline: rather than generating tasks and hoping they are solvable, we first let a strong LLM explore real APIs along graph-guided DAG structures, then synthesize tasks backward from observed outcomes, guaranteeing label correctness by construction. To handle the scale of real-world tool spaces (${\sim}$1,000 tools), we build a pairwise tool graph and sample sub-DAGs to focus exploration on semantically coherent workflows. To address environment drift in live APIs, we construct a retrieval-augmented simulator that caches all exploration results and replays them during training and evaluation, enabling fully offline and reproducible RL. Applying this pipeline yields 5,144 verified tasks spanning 240 servers and 993 tools. A 4B-parameter model trained with GRPO on FireFly matches Claude Sonnet 4.6 on our held-out test set and shows improvements on multiple tool-calling benchmarks including Tau2-Bench, MCPMark, and MCP-Atlas.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。