超越函数调用：工具环境不可靠性下的工具使用智能体基准测试
Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
AI总结：
提出ToolBench-X基准，在工具环境中注入五种可恢复可靠性风险，评估智能体任务完成能力，发现可靠工具下表现良好的智能体在可恢复风险下失败，失败主因是诊断和恢复能力不足。
摘要：
大型语言模型越来越多地被部署为智能体，通过与外部工具环境交互来解决问题。尽管最近的工具使用基准测试日益涵盖复杂的任务设置，但它们仍然在很大程度上假设干净、稳定和可信赖的工具环境，对工具环境的不可靠性考察不足。我们引入了ToolBench-X，一个用于在可恢复可靠性风险下评估智能体的基准。ToolBench-X包含跨多个领域的可执行多步任务，以及顺序、并行和混合工作流，每个任务都配有确定性工具和用于自动评估的规范最终答案。从干净的工具环境开始，ToolBench-X注入了五种结构化风险类型：规范漂移、调用错误、执行失败、输出漂移和跨源冲突。关键的是，每个注入的实例至少通过一条有效的恢复路径（如重试、回退、验证或交叉检查）仍然可解。实验揭示了显著的可靠性差距：在可靠工具下表现良好的智能体在可恢复风险下经常失败。进一步分析表明，失败更多是由于有限的风险诊断和无效的恢复，而不是工具使用量或推理预算。针对性的恢复提示可以恢复许多失败任务，而测试时缩放带来的收益有限。这些结果表明，工具使用评估应从函数调用准确性转向在不可靠工具环境下的任务完成。代码和数据可在该https URL获取。
英文摘要：
Large language models are increasingly deployed as agents that solve tasks by interacting with external tool environments. Although recent tool-use benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
