AgentEval: 基于DAG结构的步骤级评估用于具有错误传播跟踪的代理工作流
AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
AI总结：
AgentEval通过构建评估有向无环图，实现对代理工作流中错误传播的精准追踪，提升故障检测召回率和根本原因准确性，适用于生产环境中的代理系统评估。
摘要：
代理系统通过将推理、工具使用和合成串联成多步骤工作流进入生产阶段，但现有评估方法如端到端结果检查和随机跟踪检查会系统性地掩盖主导现实世界错误预算的中间故障。我们提出了AgentEval框架，将代理执行形式化为评估有向无环图（DAG），其中每个节点携带类型化的质量指标，由校准的LLM法官（GPT-4o）评估，通过分层故障分类学（3级，21个子类别）分类，并链接到上游依赖项以实现自动化根本原因归因。消融研究隔离了基于DAG的依赖建模的影响：它单独贡献了22个百分点的故障检测召回率提升和34个百分点的根本原因准确性提升，与平铺步骤级评估相比。在三个生产工作流（450个测试用例，两个代理模型家族，主要为顺序架构，其中12%的跟踪不是DAG）中，AgentEval的故障检测召回率比端到端评估高2.17倍（0.89 vs. 0.41），与人类专家的Cohen's kappa=0.84一致，且72%的根本原因准确性优于81%的人类上限。跨系统评估在tau-bench和SWE-bench跟踪上确认了可转移性（故障检测召回率≥0.78）而无需修改分类学或评分标准。一个为期四个月的试点项目中，18名工程师通过CI/CD集成的回归测试检测到23个预发布回归，将中位根本原因识别时间从4.2小时减少到22分钟，并在两个工作流中实现了可衡量的故障率降低。
英文摘要：
Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices like end-to-end outcome checks and ad-hoc trace inspection systematically mask the intermediate failures that dominate real-world error budgets. We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs), where each node carries typed quality metrics assessed by a calibrated LLM judge (GPT-4o), classified through a hierarchical failure taxonomy (3 levels, 21 subcategories), and linked to upstream dependencies for automated root cause attribution. An ablation study isolates the impact of DAG-based dependency modeling: it alone contributes +22 percentage points to failure detection recall and +34 pp to root cause accuracy over flat step-level evaluation with identical judges and rubrics. Across three production workflows (450 test cases, two agent model families, predominantly sequential architectures with a 12% non-DAG trace rate), AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation (0.89 vs. 0.41), Cohen's kappa = 0.84 agreement with human experts, and 72% root cause accuracy against an 81% human ceiling. Cross-system evaluation on tau-bench and SWE-bench traces confirms transferability (failure detection recall >= 0.78) without taxonomy or rubric modification. A 4-month pilot with 18 engineers detected 23 pre-release regressions through CI/CD-integrated regression testing, reducing median root-cause identification time from 4.2 hours to 22 minutes and driving measurable failure rate reductions in two workflows.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
