SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
The Hong Kong Polytechnic University(香港理工大学);Tencent Security Keen Lab(腾讯安全科恩实验室);Tencent Zhuque Lab(腾讯朱雀实验室);Tencent Security Platform and Department(腾讯安全平台部)
Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, Xiapu Luo
Evaluating Large Language Models (LLMs) is crucial for understanding their capabilities and limitations across various applications, including natural language processing and code generation. Existing benchmarks like MMLU, C-Eval, and HumanEval assess general LLM performance but lack focus on specific expert domains such as cybersecurity. Previous attempts to create cybersecurity datasets have faced limitations, including insufficient data volume and a reliance on multiple-choice questions (MCQs). To address these gaps, we propose SecBench, a multi-dimensional benchmarking dataset designed to evaluate LLMs in the cybersecurity domain. SecBench includes questions in various formats (MCQs and short-answer questions (SAQs)), at different capability levels (Knowledge Retention and Logical Reasoning), in multiple languages (Chinese and English), and across various sub-domains. The dataset was constructed by collecting high-quality data from open sources and organizing a Cybersecurity Question Design Contest, resulting in 44,823 MCQs and 3,087 SAQs. Particularly, we used the powerful while cost-effective LLMs to (1). label the data and (2). constructing a grading agent for automatic evaluation of SAQs. Benchmarking results on 16 SOTA LLMs demonstrate the usability of SecBench, which is arguably the largest and most comprehensive benchmark dataset for LLMs in cybersecurity. More information about SecBench can be found at our website, and the dataset can be accessed via the artifact link.
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Department of Cybersecurity(网络安全系);Department of Information Sciences and Technology(信息科学与技术系);College of Emergency Preparedness, Homeland Security, and Cybersecurity(应急准备、国土安全与网络安全学院);University at Albany, SUNY(阿尔巴尼大学)
Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence
Technology Innovation Institute(技术创新研究院);University of Oslo(奥斯陆大学);Eötvös Loránd University(厄特沃什·罗兰大学);University of Guelma(盖勒马大学);The University of Manchester(曼彻斯特大学);Khalifa University(哈利法大学);Cyentific AS(赛恩蒂菲克公司)
Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
School of Computer Science and Mathematics, Keele University(基尔大学计算机科学与数学学院);Cybersecurity Institute, University of Liverpool(利物浦大学网络安全研究所);Department of Applied Computing IICL, University of Wales Trinity Saint David(威尔士特里尼达大学应用计算系);Deakin Cyber Research and Innovation Hub, Deakin University(德金大学网络安全研究与创新中心);Department of Information Systems and Cyber Security, The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校信息系与网络安全系)