arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21098cs.CLcs.AIcs.LGcs.SE
收藏

ToolBottleneckBench:在受控压力和固定工具调用框架预算下测量工具智能体性能

ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

University of Osaka(大阪大学);Initial S;Nagoya University(名古屋大学)

Zichuan Xu, Yuyang Dong, Makoto Onizuka, Chuan Xiao

AI总结:

本研究提出ToolBottleneckBench基准,在受控压力和固定工具调用框架预算下测量工具智能体性能,发现可见工具菜单是对模型产生较大惩罚的关键压力因素,提高完成上限可缓解该惩罚。

摘要:

工具智能体的成功得分既反映了其模型性能,也反映了运行该智能体的工具调用框架(harness)的预算情况。我们提出了ToolBottleneckBench,这是一个在受控压力和固定工具调用框架预算下测量工具智能体性能的基准。合成企业任务保持固定,同时改变四种压力因素中的一种:注册的工具目录、可见的工具菜单、认证的工作流或返回记录的编码。精确评分器会记录每个失败回合中第一个未满足的要求,以及导致该回合结束的工具调用框架限制(若存在)。在21种模型和工具调用框架配置中,三种压力因素里只有可见菜单会对模型产生较大惩罚,且该惩罚与完成上限(completion cap)的截断情况相关。将上限提高16倍,可消除大部分以截断为主的配置中的惩罚。大多数失败在工具调用框架限制处结束,且失败记录识别分配的压力因素的可靠性并不比工具调用次数计数更高。有助于改变哪些工具到达模型或谁发起调用的修复措施,以及仅两种均未改善模型行为的修复措施,胜过所有配置切换。我们建议报告压力因素效应时,需注明测量时所用的工具调用框架预算。我们在该https URL发布了此基准。

英文摘要:

The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark that measures tool-agent performance under controlled pressures and fixed harness budgets. Synthetic enterprise tasks stay fixed while one pressure varies: the registered tool catalog, the visible tool menu, the certified workflow, or the encoding of returned records. An exact grader records the first unmet obligation of each failed episode and the harness limit, if any, that ended it. Across 21 model and harness configurations, the visible menu is the only one of the three pressures that act on the model with a large penalty, and this penalty tracks truncation at the completion cap. Raising the cap sixteenfold removes most of the penalty in the truncation-heavy configurations. Most failures end at a harness limit, and the failure record identifies the assigned pressure no more reliably than a count of tool calls. The repairs that help change which tools reach the model or who issues the calls, and only two repairs, neither improving model behavior, beat every configuration switch. We recommend reporting pressure effects with the harness budgets under which they were measured. We release the benchmark at https://github.com/zichuanxu/tool-bottleneck-bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑