arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2611.06099cs.SE
收藏

SWEOps-Bench:评估端到端解决软件引发的云事件的智能体

SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud Incidents

Sun Yat-sen University(中山大学);The Chinese University of Hong Kong(香港中文大学)

Danwen Chen, Guangba Yu, Pengfei Chen

AI总结:

SWEOps-Bench是含55个Kubernetes Operator相关事件任务的交互式基准,评估6种智能体-模型配置的事件解决能力,发现其端到端恢复率远低于故障检测率,为相关智能体评估提供了严格基础。

摘要:

随着自主智能体的发展,软件开发与云运维之间的传统边界逐渐模糊,对集成式事件检测、定位和修复的需求日益增长。现有基准仅评估孤立的诊断查询或仓库级别的程序修复,无法测试跨运维集群与源代码仓库的连续事件解决。我们推出SWEOps-Bench,这是一个交互式基准,包含55个可复现的事件任务,源自16个Kubernetes Operator项目中已确认的历史缺陷。每个任务保留原始软件仓库,并在受控工作负载触发历史故障前初始化健康集群;智能体需调查运行时证据,通过终端界面提交源代码或配置修复,测试工具则在共享时间预算内构建并部署变更。评估涵盖阶段准确率、已验证的恢复结果、受惩罚的生命周期延迟以及令牌消耗。在6种智能体-模型配置下,故障状态检测准确率为72.7%至90.9%,而整体恢复率仅为14.5%至52.7%,暴露了识别事件与端到端解决事件之间存在巨大差距。轨迹分析进一步显示,报告的诊断结果与最终恢复结果仅部分重合:部分任务虽报告的根本原因分析(RCA)不正确,但仍实现了已验证的恢复;而运行时症状的消除仍可能无法通过修复特定的功能验证。SWEOps-Bench为评估集成式开发与运维环境中的自主智能体奠定了严格的基础。

英文摘要:

As autonomous agents advance, the traditional boundary between software development and cloud operations gradually blurs, increasing the need for integrated incident detection, localization, and remediation. Existing benchmarks evaluate isolated diagnostic queries or repository-level program repair, failing to test continuous incident resolution across operational clusters and source repositories. We introduce SWEOps-Bench, an interactive benchmark comprising 55 reproducible incident tasks derived from confirmed historical defects across 16 Kubernetes operator projects. Each task preserves the original software repository and initializes in a healthy cluster before a controlled workload triggers the historical fault. Agents investigate runtime evidence and submit source or configuration repairs through a terminal interface, while the harness builds and deploys the changes within a shared time budget. Evaluation covers stage-wise accuracy, verified recovery, penalized lifecycle latency, and token consumption. Across six agent-model configurations, faulty-state detection ranges from 72.7% to 90.9%, whereas overall recovery ranges from only 14.5% to 52.7%, exposing a substantial gap between recognizing incidents and resolving them end to end. Trajectory analysis further shows that reported diagnosis and final recovery only partially coincide: some tasks achieve verified recovery despite an incorrect reported RCA, while runtime symptom clearance can still fail fix-specific functional verification. SWEOps-Bench establishes a rigorous foundation for evaluating autonomous agents in integrated development and operational environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑