Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, agents exhibit distinct attack-selection biases, differing in both the families they explore and the concentration of their efforts. These patterns remain relatively stable across prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evaluates five agents on three targets and four prompt conditions with ten attack families. We identify explicit bias across agents, with different dominant attack families and varying entropy levels in their attack-family allocation distributions. Such bias characterizes agent behavior within the evaluated environments and provides information beyond aggregate attack success rates. Furthermore, our experiments reveal a bias momentum effect, where agents resist explicit steering toward attack families that conflict with their bias. This forced distribution shift does not yield measurable improvements in attack performance. To ensure reproducibility and facilitate future research, we release an interactive result dashboard at https://trustworthyai.co.kr/CyBiasBench/ and a reproducibility artifact with aggregated session-level statistics and full evaluation scripts at https://github.com/Harry24k/CyBiasBench.
SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
The Hong Kong Polytechnic University(香港理工大学);Tencent Security Keen Lab(腾讯安全科恩实验室);Tencent Zhuque Lab(腾讯朱雀实验室);Tencent Security Platform and Department(腾讯安全平台部)
本文提出网络安全多维基准数据集 SecBench,覆盖多题型、能力层级、语言与子领域,并借助 LLM 完成标注和 SAQ 自动评分,在 16 个 SOTA 模型上验证了其可用性。
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Department of Cybersecurity(网络安全系);Department of Information Sciences and Technology(信息科学与技术系);College of Emergency Preparedness, Homeland Security, and Cybersecurity(应急准备、国土安全与网络安全学院);University at Albany, SUNY(阿尔巴尼大学)
GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks
Cyber Security and Resilience Technology (CyberSaR), King Abdullah University of Science and Technology (KAUST)(网络安全与韧性技术(CyberSaR),国王阿卜杜勒·阿齐兹大学科学与技术学院(KAUST))
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
Technology Innovation Institute(技术创新研究院);University of Oslo(奥斯陆大学);Eötvös Loránd University(厄特沃什·罗兰大学);University of Guelma(盖勒马大学);The University of Manchester(曼彻斯特大学);Khalifa University(哈利法大学);Cyentific AS(赛恩蒂菲克公司)
Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
School of Computer Science and Mathematics, Keele University(基尔大学计算机科学与数学学院);Cybersecurity Institute, University of Liverpool(利物浦大学网络安全研究所);Department of Applied Computing IICL, University of Wales Trinity Saint David(威尔士特里尼达大学应用计算系);Deakin Cyber Research and Innovation Hub, Deakin University(德金大学网络安全研究与创新中心);Department of Information Systems and Cyber Security, The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校信息系与网络安全系)
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
AI Open Source Foundation(2077AI开源基金会);UC Santa Cruz(加州大学圣克鲁兹分校);UC Berkeley(加州大学伯克利分校);UC Santa Barbara(加州大学圣巴巴拉分校);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Massachusetts Institute of Technology(麻省理工学院);University of Toronto(多伦多大学);Johns Hopkins University(约翰斯·霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学);Purdue University(普渡大学);Texas A&M University(德克萨斯农工大学)
ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of System Software (Chinese Academy of Sciences)(中国科学院系统软件重点实验室)