AI 中文总结
针对工具检索基准中一对一标注低估检索器的问题，提出ToolEX框架自动扩展等价工具组合，证明30-47%的微调增益是评估假象。
AI 中文摘要
在具有大规模且不断演化的工具库的开放世界场景中，工具增强型大语言模型依赖检索器为给定查询找出相关工具。由于此类工具库通常包含许多实现相同功能的工具，单个查询往往可由多个不同但功能等价的工具组合来解决，这使得自然的查询到工具的映射本质上是一对多的。然而，现有的工具检索基准为每个查询仅标注一个相关的工具组合，将这种一对多映射压缩为严格的一对一标注，导致有效的检索工具被误判为失败。为解决此问题，我们提出ToolEX（工具等价扩展），一个自动发现并标注与已标注组合功能等价的工具组合的框架。应用于包含7,360个查询的Tool-DE基准时，ToolEX发现67.9%的子查询存在等价替代方案，将单一真值扩展为每个查询平均5.3个有效组合。使用扩展后的基准ToolEQ，我们重新评估了八个基础检索器和两个微调变体；ToolEQ上的指标相比Tool-DE显著提升，表明一对一标注系统性地低估了检索器，且所报告的微调增益中有30%至47%是评估假象而非真正的改进。将相同流程应用于SkillRet上的技能检索，进一步证实一对一问题不仅限于工具检索。
英文摘要
In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.
