Who&When Pro：大语言模型真的能归因人工智能代理中的失败吗？
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
浏览论文内容
中文总结 AI 辅助
研究聚焦于大语言模型能否归因人工智能代理中的失败，引入Who&When Pro基准，通过严格管道构建大量失败轨迹，经广泛实验分析，揭示模型归因故障模式，为自动故障归因系统提供实证指导。
中文摘要 AI 辅助
自动故障归因利用大语言模型来识别代理系统故障的位置和原因。随着代理能力增强，其故障更难察觉，自动归因愈发重要。我们引入Who&When Pro，一个用于代理系统自动故障归因的大规模基准。通过严格控制的管道，在精确重放成功前缀后注入故障，构建了12326条带黄金标签的失败轨迹，涵盖3种模态和26个基准。除基准测试外，还进行了广泛实验与分析，揭示了模型跨模态、协议和模型家族归因故障的系统模式，并为未来自动故障归因系统提供了实证指导。
英文摘要
Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.
发表机构
- OpenAI
- Anthropic
- Google DeepMind（谷歌DeepMind）
机构由 AI 辅助整理，请以论文原文为准。
