arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01508cs.SEcs.AI
收藏

ToolMisuseBench: 一个用于代理系统中工具误用和恢复的离线确定性基准

ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems

Independent Researcher(独立研究员)

Akshey Sigdel, Rista Baral

AI总结:

本文提出ToolMisuseBench,用于评估代理系统中工具误用和恢复的性能,包含6800个任务和可复现的评估流程,展示了基于模式意识方法的故障恢复优势。

摘要:

使用工具的代理在操作原因下往往失败,即使语言理解能力强。常见原因包括无效参数、界面漂移、弱恢复和低效重试行为。我们引入ToolMisuseBench,一个离线确定性基准,用于评估在显式步骤、调用和重试预算下的工具误用和恢复。该基准涵盖CRUD、检索、文件和调度环境,并具有可重放的故障注入。它报告成功、无效调用行为、政策违规、恢复质量及预算效率。我们发布了一个包含6800个任务的公开数据集和可复现的评估流程。基准结果表明,基于模式意识的方法在特定故障恢复方面有所提升,但在发布的授权和硬故障设置下整体成功率仍有限。

英文摘要:

Tool using agents often fail for operational reasons even when language understanding is strong. Common causes include invalid arguments, interface drift, weak recovery, and inefficient retry behavior. We introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and recovery under explicit step, call, and retry budgets. The benchmark covers CRUD, retrieval, file, and scheduling environments with replayable fault injection. It reports success, invalid call behavior, policy violations, recovery quality, and budgeted efficiency. We release a public dataset with 6800 tasks and a reproducible evaluation pipeline. Baseline results show fault specific recovery gains for schema aware methods, while overall success remains limited under the released authorization and hard failure settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑