CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking
CISPA - Helmholtz Center for Information Security（CISPA-海德堡信息安全研究中心）；NYU Tandon School of Engineering（纽约大学坦顿工程学院）；NYU Abu Dhabi（纽约大学阿布扎克分校）；IIIT Hyderabad（海得拉巴印度理工学院）
Existing benchmarks for LLM-based offensive security agents use isolated, single-target setups with a known vulnerable service and fixed objective. They measure exploitation effectively, but miss how real Capture-the-Flag (CTF) participants triage unknown surfaces, prioritize targets, and allocate effort under uncertainty. Current evaluations therefore fail to assess strategic reasoning beyond exploitation alone. To address this, we introduce \textit{CTFExplorer}, a benchmark suite that shifts offensive security evaluation toward a multi-target setting, which tests how agents explore, prioritize, and chain attacks. CTFExplorer deploys 40 web-based vulnerable services within a single environment, where agents must autonomously discover, distinguish, and exploit targets without predefined guidance. We also present a reactive multi-agent setup as a reference agent framework and develop an agent-agnostic evaluation framework that records structured reasoning traces for fine-grained assessment. This enables behavioral evaluation beyond binary flag capture, such as how agents manage target selection, handle failed hypotheses, coordinate across multiple stages, and extract security intelligence.
Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
New York University（纽约大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；International Institute of Information Technology Hyderabad（国际信息技术学院海得拉巴分校）
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Department of Cybersecurity（网络安全系）；Department of Information Sciences and Technology（信息科学与技术系）；College of Emergency Preparedness, Homeland Security, and Cybersecurity（应急准备、国土安全与网络安全学院）；University at Albany, SUNY（阿尔巴尼大学）
How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation
NYU Tandon School of Engineering（纽约大学坦登工程学院）；NYU Abu Dhabi（纽约大学阿布扎比分校）；CISPA - Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；International Institute of Information Technology Hyderabad（海得拉巴国际信息技术学院）；Indian Institute of Technology Tirupati（蒂鲁帕蒂印度理工学院）
Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
Alias Robotics；Università degli Studi di Napoli Federico II（那不勒斯费德里科二世大学）；Johannes Kepler University Linz（林茨约翰内斯开普勒大学）；Alpen-Adria-Universität Klagenfurt（阿尔卑斯-亚得里亚-克拉根福大学）
该研究利用 CAI 框架在 23 个攻防 CTF 中实测自主 AI 智能体，发现防御优势仅在无约束补丁条件下成立，实际可用性与防入侵约束会显著拉低其表现。
VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents
Department of Computer Science Tennessee Tech University Cookeville, TN, USA；School of Interdisciplinary Informatics University of Nebraska Omaha Omaha, NE, USA；1 Department of Computer Science, Tennessee Tech University, TN, USA.；2 School of Interdisciplinary Informatics, University of Nebraska Omaha, NE, USA.；3 Dept. of Mathematics \& Computer Science, University of North Carolina at Pembroke, NC, USA.
