为何AI代理在云根因分析中系统性失败?
Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?
AI总结:
本文分析了基于LLM的云根因分析代理在推理、通信和环境交互中的系统性故障,揭示了幻觉数据和不完全探索等普遍问题,并通过改进通信协议减少故障率。
摘要:
大规模云系统故障会造成巨大经济损失,因此自动化根因分析(RCA)对运营稳定性至关重要。近期研究利用大语言模型(LLM)代理自动化此任务,但现有系统即使在有能力的模型下也表现出低检测准确率,且当前评估框架仅评估最终答案的正确性,而无法揭示代理推理失败的原因。本文对基于LLM的RCA代理进行了过程层面的故障分析。我们执行了五个LLM模型的完整OpenRCA基准测试,生成了1,675次代理运行,并将观察到的故障分类为12种类型,涵盖代理内部推理、代理间通信以及代理与环境交互。我们的分析表明,最普遍的故障,如幻觉数据解释和不完全探索,在所有模型中普遍存在,表明这些故障源于共享的代理架构,而非单个模型的限制。受控缓解实验进一步表明,仅通过提示工程无法解决主要故障,而丰富代理间通信协议可将与通信相关的故障减少多达15个百分点。本文开发的故障分类法和诊断方法为设计更可靠的云RCA自主代理提供了基础。
英文摘要:
Failures in large-scale cloud systems incur substantial financial losses, making automated Root Cause Analysis (RCA) essential for operational stability. Recent efforts leverage Large Language Model (LLM) agents to automate this task, yet existing systems exhibit low detection accuracy even with capable models, and current evaluation frameworks assess only final answer correctness without revealing why the agent's reasoning failed. This paper presents a process level failure analysis of LLM-based RCA agents. We execute the full OpenRCA benchmark across five LLM models, producing 1,675 agent runs, and classify observed failures into 12 pitfall types across intra-agent reasoning, inter-agent communication, and agent-environment interaction. Our analysis reveals that the most prevalent pitfalls, notably hallucinated data interpretation and incomplete exploration, persist across all models regardless of capability tier, indicating that these failures originate from the shared agent architecture rather than from individual model limitations. Controlled mitigation experiments further show that prompt engineering alone cannot resolve the dominant pitfalls, whereas enriching the inter-agent communication protocol reduces communication-related failures by up to 15 percentage points. The pitfall taxonomy and diagnostic methodology developed in this work provide a foundation for designing more reliable autonomous agents for cloud RCA.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。