arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07688cs.AIcs.CR
收藏

CyberMetric:一个基于检索增强生成的基准数据集,用于评估大语言模型在网络安全领域的知识水平

CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge

Technology Innovation Institute (TII)(技术创新研究所(TII));University of Oslo(奥斯陆大学);Khalifa University(哈利法大学)

Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, Merouane Debbah

AI总结:

本文提出基于检索增强生成构建的网络安全多选题基准数据集CyberMetric,用于评估大语言模型在网络安全领域的通用知识,并对比了25个最先进模型与人类专家的表现。

摘要:

大语言模型(LLMs)在从软件开发到网络威胁情报等各个领域的应用日益广泛。理解网络安全的所有不同领域(包括密码学、逆向工程和风险评估等主题),即使对人类专家来说也是一项挑战。为了准确测试LLMs在网络安全方面的通用知识,研究界需要一个多样化、准确且最新的数据集。为了填补这一空白,我们提出了CyberMetric-80、CyberMetric-500、CyberMetric-2000和CyberMetric-10000,这些是分别包含80、500、2000和10000道题目的选择题问答基准数据集。通过利用GPT-3.5和检索增强生成(RAG),我们收集了网络安全领域的文档(包括NIST标准、研究论文、公开书籍、RFCs及其他出版物)来生成问题,每个问题提供四个可能的答案。这些结果经过了多轮错误检查和修正。人类专家投入了超过200小时验证问题和解决方案,以确保其准确性和相关性,并过滤掉任何与网络安全无关的问题。我们在CyberMetric数据集上评估并比较了25个最先进的LLM模型。除了评估LLMs这一主要目标外,我们还邀请了30名人类参与者在闭卷场景下解答CyberMetric-80。这些结果可作为比较人类和LLMs通用网络安全知识的参考。研究结果显示,GPT-4o、GPT-4-turbo、Mixtral-8x7B-Instruct、Falcon-180B-Chat和GEMINI-pro 1.0是表现最好的LLMs。此外,在CyberMetric-80上,顶尖的LLMs比人类更准确,尽管经验丰富的人类专家仍然优于Llama-3-8B、Phi-2或Gemma-7b等小型模型。

英文摘要:

Large Language Models (LLMs) are increasingly used across various domains, from software development to cyber threat intelligence. Understanding all the different fields of cybersecurity, which includes topics such as cryptography, reverse engineering, and risk assessment, poses a challenge even for human experts. To accurately test the general knowledge of LLMs in cybersecurity, the research community needs a diverse, accurate, and up-to-date dataset. To address this gap, we present CyberMetric-80, CyberMetric-500, CyberMetric-2000, and CyberMetric-10000, which are multiple-choice Q&A benchmark datasets comprising 80, 500, 2000, and 10,000 questions respectively. By utilizing GPT-3.5 and Retrieval-Augmented Generation (RAG), we collected documents, including NIST standards, research papers, publicly accessible books, RFCs, and other publications in the cybersecurity domain, to generate questions, each with four possible answers. The results underwent several rounds of error checking and refinement. Human experts invested over 200 hours validating the questions and solutions to ensure their accuracy and relevance, and to filter out any questions unrelated to cybersecurity. We have evaluated and compared 25 state-of-the-art LLM models on the CyberMetric datasets. In addition to our primary goal of evaluating LLMs, we involved 30 human participants to solve CyberMetric-80 in a closed-book scenario. The results can serve as a reference for comparing the general cybersecurity knowledge of humans and LLMs. The findings revealed that GPT-4o, GPT-4-turbo, Mixtral-8x7B-Instruct, Falcon-180B-Chat, and GEMINI-pro 1.0 were the best-performing LLMs. Additionally, the top LLMs were more accurate than humans on CyberMetric-80, although highly experienced human experts still outperformed small models such as Llama-3-8B, Phi-2 or Gemma-7b.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑