arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.32285cs.AI
收藏

通过轨迹测试:对AI智能体管控工具的可靠性与失败归因进行基准测试

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Institute of Automation, CAS(中国科学院自动化研究所);Zhongguancun Academy(中关村学院);Institute of Information Engineering, CAS(中国科学院信息工程研究所);Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学)
Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-L… 展开作者

Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou

AI总结:

该研究针对现有智能体管控工具评估不足的问题,推出TraceTrial-Bench基准,设计TT、MEA、FA模块,对4种管控工具、9个模型的95项任务开展评估,为管控工具比较与失败诊断提供基础。

摘要:

大语言模型(LLM)智能体越来越多地被用于解决现实世界问题,其有效性既取决于底层模型,也取决于支撑其运行的管控工具(harness)。然而,大多数现有基准测试聚焦于模型能力和智能体整体性能,对智能体管控工具的评估关注有限。为填补这一空白,我们推出TraceTrial-Bench,该基准将管控工具的可靠性划分为7个组件,涵盖上下文、执行、治理、生命周期、可观测性、工具和验证。我们构建了95个专家级、面向机制的任务,这些任务具有可执行或基于状态的真实值,以支持语义等价的比较,同时保留每个管控工具的原生行为。我们设计了轨迹追踪(TT)模块来捕获智能体执行轨迹,并引入了两层评估框架,分别评估智能体性能和管控工具可靠性。此外,我们设计了多源证据获取(MEA)模块,从智能体上下文、管控工具和执行环境中提供证据,用于基于大语言模型的评估,并开发了失败归因(FA)模块,该模块使用显式机制链来支持运行级别的归因。在对4种管控工具、9个模型和95个任务的评估中,我们对3420次运行的评估得出,管控工具的宏平均得分在59.8%至72.0%之间,而31.1%的运行尽管未满足至少一项管控工具标准,但仍实现了任务成功。TraceTrial-Bench为比较智能体管控工具、诊断其失败以及研究管控工具设计与模型行为的相互作用提供了可审计且可复现的基础。

英文摘要:

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑