Large Language Models (LLMs) are increasingly used across various domains, from software development to cyber threat intelligence. Understanding all the different fields of cybersecurity, which includes topics such as cryptography, reverse engineering, and risk assessment, poses a challenge even for human experts. To accurately test the general knowledge of LLMs in cybersecurity, the research community needs a diverse, accurate, and up-to-date dataset. To address this gap, we present CyberMetric-80, CyberMetric-500, CyberMetric-2000, and CyberMetric-10000, which are multiple-choice Q&A benchmark datasets comprising 80, 500, 2000, and 10,000 questions respectively. By utilizing GPT-3.5 and Retrieval-Augmented Generation (RAG), we collected documents, including NIST standards, research papers, publicly accessible books, RFCs, and other publications in the cybersecurity domain, to generate questions, each with four possible answers. The results underwent several rounds of error checking and refinement. Human experts invested over 200 hours validating the questions and solutions to ensure their accuracy and relevance, and to filter out any questions unrelated to cybersecurity. We have evaluated and compared 25 state-of-the-art LLM models on the CyberMetric datasets. In addition to our primary goal of evaluating LLMs, we involved 30 human participants to solve CyberMetric-80 in a closed-book scenario. The results can serve as a reference for comparing the general cybersecurity knowledge of humans and LLMs. The findings revealed that GPT-4o, GPT-4-turbo, Mixtral-8x7B-Instruct, Falcon-180B-Chat, and GEMINI-pro 1.0 were the best-performing LLMs. Additionally, the top LLMs were more accurate than humans on CyberMetric-80, although highly experienced human experts still outperformed small models such as Llama-3-8B, Phi-2 or Gemma-7b.
Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generation
School of Electrical and Computer Engineering(电气与计算机工程学院);Georgia Institute of Technology(佐治亚理工学院);Department of Computer Science(计算机科学系);Rochester Institute of Technology(罗切斯特理工学院)
Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens
SSH Communications Security(SSH通讯安全公司);KTH Royal Institute of Technology(皇家理工学院);Aalto University(阿尔托大学)
MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation
Istituto di Informatica e Telematica, Consiglio Nazionale Delle Ricerche(国家研究委员会信息与自动化研究所);TeCIP, Scuola Universitaria Superiore Sant’Anna(圣安娜高等学校TeCIP研究所);University of Padua(帕多瓦大学);Delft University of Technology(代尔夫特理工大学)
CyberBOT: Towards Reliable Cybersecurity Education via Ontology-Grounded Retrieval Augmented Generation
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);Mary Lou Fulton Teachers College, Arizona State University(亚利桑那州立大学玛丽·卢·富尔顿师范学院)
Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Alabama Transportation Institute, The University of Alabama(阿拉巴马大学阿拉巴马交通研究所);The University of Alabama(阿拉巴马大学);Clemson University(克莱姆森大学)
Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy
Digital Transit Limited, 3M Buckley Innovation Centre(数字交通有限公司,3M Buckley创新中心);Department of Computer Science at University of Huddersfield(赫德瑟尔大学计算机科学系)
Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities
Guelma University(盖勒马大学);Technology Innovation Institute(技术创新研究院);Concordia University(康考迪亚大学);University of Oslo(奥斯陆大学);Khalifa University of Science and Technology(哈利法科技大学)
Leveraging Large Language Models for Cybersecurity Risk Assessment -- A Case from Forestry Cyber-Physical Systems
Chalmers University of Technology(楚德斯技术大学);University of Gothenburg(哥德堡大学);Carnegie Mellon University(卡内基梅隆大学);RISE Research Institutes of Sweden(瑞典RISE研究机构)
RAG-PRISM: A Personalized, Rapid, and Immersive Skill Mastery Framework with Adaptive Retrieval-Augmented Tutoring
College of Information Science, University of Arizona, Tucson, AZ, USA(信息科学学院,亚利桑那大学);Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA(电气与计算机工程系,亚利桑那大学);Department of Systems and Industrial Engineering, University of Arizona, Tucson, AZ, USA(系统与工业工程系,亚利桑那大学);Department of Industrial Engineering, University of Sonora, Hermosillo, Mexico(工业工程系,索诺拉大学);Leonhard Center for Enhancement of Engineering Education, The Pennsylvania State University, University Park, PA, USA(工程教育增强中心,宾夕法尼亚州立大学)
AMAQA: A Metadata-based QA Dataset for RAG Systems
Institute for Informatics and Telematics, National Research Council(信息与电信研究院,国家研究院);Department of Information Engineering, University of Pisa(信息工程系,比萨大学);Department of Computer Science, University of Pisa(计算机科学系,比萨大学)
Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions
School of Computer Science and Mathematics, Keele University(基尔大学计算机科学与数学学院);Cybersecurity Institute, University of Liverpool(利物浦大学网络安全研究所);Department of Applied Computing IICL, University of Wales Trinity Saint David(威尔士特里尼达大学应用计算系);Deakin Cyber Research and Innovation Hub, Deakin University(德金大学网络安全研究与创新中心);Department of Information Systems and Cyber Security, The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校信息系与网络安全系)
OntoLogX: Ontology-Guided Knowledge Graph Extraction from Cybersecurity Logs with Large Language Models
Department of Information Engineering, University of Brescia, Italy(博洛尼亚大学信息工程系,意大利);Department of Computer Science, University of Turin, Italy(都灵大学计算机科学系,意大利);School of Computer Science and Informatics, Cardiff University, United Kingdom(卡迪夫大学计算机科学与信息学学院,英国);Department of Electronics and Computer Science, University of Southampton, United Kingdom(南安普顿大学电子与计算机科学系,英国)