Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.
Economics of Human and AI Collaboration: When is Partial Automation More Attractive than Full Automation?
Massachusetts Institute of Technology(麻省理工学院);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院);IBM Research(IBM研究院);IBM’s Institute for Business Value(IBM商业价值研究院)
University of Sydney(悉尼大学);University of Adelaide(阿德莱德大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学)
DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice
Product Development Division, Department of Mechanical Engineering, Chalmers University of Technology, Gothenburg, Sweden(产品开发部门,机械工程系,楚姆勒斯技术大学,哥德堡,瑞典);GKN Aerospace Engine Systems, Trollhättan, Sweden(GKN航空发动机系统,特罗尔哈塔纳,瑞典)
HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences(自动化研究所人工智能与自然语言处理实验室,中国科学院);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Meituan(美团)
CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamics
Rensselaer Polytechnic Institute(拉特格斯理工学院);University of California, San Diego(加州大学圣地亚哥分校);National Laboratory of the Rockies(落基山国家实验室);Pacific Northwest National Laboratory(太平洋西北国家实验室)
A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites
Polytechnic University of Bari \& Wideverse Bari, Italy;Polytechnic University of Bari Bari, Italy;Independent Researcher;University of Bari Bari, Italy
PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design
School of Computing and Information Systems, Singapore Management University(计算与信息系统学院,新加坡国立管理大学);School of Computer Science and Engineering, Nanyang Technological University(计算机科学与工程学院,南洋理工大学);School of Cyber Security, Tianjin University(网络安全学院,天津大学)