arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.05352cs.AIcs.DCcs.MA
收藏

ITBench:评估AI智能体在多样化现实世界IT自动化任务中的表现

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

IBM(国际商业机器公司);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutl… 展开作者

Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir O. Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, Pavankumar Murali, Jae-wook Ahn, Debanjana Kar, Ameet Rahane, Carlos Fonseca, Amit Paradkar, Yu Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav R. Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas C. M. Fuller, Ruchir Puri

AI总结:

本文提出ITBench框架,为AI智能体在SRE、CISO、FinOps等现实IT自动化任务提供基准测试方法,含94个场景,助力研究者理解挑战,推动AI驱动IT自动化发展。

摘要:

利用AI智能体实现关键IT任务自动化的愿景,取决于衡量和理解所提方案有效性的能力。我们引入ITBench框架,为基准测试AI智能体以解决现实世界IT自动化任务提供系统化方法。其初始版本针对三个关键领域:站点可靠性工程(SRE)、合规与安全运营(CISO)及财务运营(FinOps)。该设计通过一键式工作流和可解释指标,让AI研究者了解AI智能体在IT自动化中的挑战与机遇。ITBench包含94个现实场景初始集,可通过社区贡献轻松扩展。结果显示,由最先进模型驱动的智能体仅解决13.8%的SRE场景、25.2%的CISO场景和0%的FinOps场景。我们期望ITBench能成为推动正确、安全且快速的AI驱动IT自动化的关键工具。

英文摘要:

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑