Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
Institute of Automation, CAS(中国科学院自动化研究所);Zhongguancun Academy(中关村学院);Institute of Information Engineering, CAS(中国科学院信息工程研究所);Tsinghua University(清华大学);Beijing Institute of Technology(北京理工大学)
LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.
SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills
The Ohio State University(俄亥俄州立大学);The University of Chicago(芝加哥大学);University College London(伦敦大学学院);University of Michigan(密歇根大学);The Chinese University of Hong Kong(香港中文大学);Case Western Reserve University(凯斯西储大学);Amazon(亚马逊)
SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
Zhejiang University(浙江大学);Ant Digital Technologies, Ant Group(蚂蚁集团数字技术部);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室)
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Peking University School of Software and Microelectronics(北京大学软件与微电子学院);Peking University School of Psychological and Cognitive Sciences(北京大学心理与认知科学学院);Key Laboratory of Machine Perception (Ministry of Education), Peking University(北京大学机器感知重点实验室)
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Tsinghua University(清华大学);Peking University(北京大学);Southeast University(东南大学)
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学)
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
University of California, Berkeley(加州大学伯克利分校);Purdue University(普渡大学);University of Ljubljana(卢布尔雅那大学);University of Washington(华盛顿大学);Oasis Labs;University of Maryland(马里兰大学);IBM Research(IBM研究院);Mila;McGill University(麦吉尔大学);ServiceNow Research(ServiceNow研究院);Carnegie Mellon University(卡内基梅隆大学);National Institute of Standards and Technology(美国国家标准与技术研究院);The Ohio State University(俄亥俄州立大学);University of Cambridge(剑桥大学);University of California, Santa Barbara(加州大学圣塔芭芭拉分校)