arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.20787cs.CRcs.AI
收藏

SecBench:面向网络安全领域大语言模型的综合多维基准评测数据集

SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity

The Hong Kong Polytechnic University(香港理工大学);Tencent Security Keen Lab(腾讯安全科恩实验室);Tencent Zhuque Lab(腾讯朱雀实验室);Tencent Security Platform and Department(腾讯安全平台部)

Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, Xiapu Luo

AI总结:

本文提出网络安全多维基准数据集 SecBench,覆盖多题型、能力层级、语言与子领域,并借助 LLM 完成标注和 SAQ 自动评分,在 16 个 SOTA 模型上验证了其可用性。

摘要:

评估大语言模型(LLM)对于理解其在自然语言处理、代码生成等各类应用中的能力与局限至关重要。MMLU、C-Eval 和 HumanEval 等现有基准主要评估 LLM 的通用性能,却缺乏对网络安全等特定专家领域的关注。以往构建网络安全数据集的尝试存在数据量不足、依赖多项选择题(MCQ)等局限。为弥补这些差距,我们提出 SecBench,这是一个用于评估 LLM 在网络安全领域表现的多维基准数据集。SecBench 包含多种题型(MCQ 与简答题(SAQ))、不同能力层级(知识记忆与逻辑推理)、多种语言(中文和英文),并覆盖多个子领域。该数据集通过收集开源高质量数据并举办网络安全题目设计竞赛构建而成,共包含 44,823 道 MCQ 和 3,087 道 SAQ。特别地,我们利用功能强大且具有成本效益的 LLM 来:(1)标注数据;(2)构建用于自动评估 SAQ 的评分智能体。在 16 个 SOTA LLM 上的基准测试结果证明了 SecBench 的可用性;它可以说是目前规模最大、最全面的面向网络安全领域 LLM 的基准数据集。有关 SecBench 的更多信息可访问我们的网站,数据集可通过 artifact 链接获取。

英文摘要:

Evaluating Large Language Models (LLMs) is crucial for understanding their capabilities and limitations across various applications, including natural language processing and code generation. Existing benchmarks like MMLU, C-Eval, and HumanEval assess general LLM performance but lack focus on specific expert domains such as cybersecurity. Previous attempts to create cybersecurity datasets have faced limitations, including insufficient data volume and a reliance on multiple-choice questions (MCQs). To address these gaps, we propose SecBench, a multi-dimensional benchmarking dataset designed to evaluate LLMs in the cybersecurity domain. SecBench includes questions in various formats (MCQs and short-answer questions (SAQs)), at different capability levels (Knowledge Retention and Logical Reasoning), in multiple languages (Chinese and English), and across various sub-domains. The dataset was constructed by collecting high-quality data from open sources and organizing a Cybersecurity Question Design Contest, resulting in 44,823 MCQs and 3,087 SAQs. Particularly, we used the powerful while cost-effective LLMs to (1). label the data and (2). constructing a grading agent for automatic evaluation of SAQs. Benchmarking results on 16 SOTA LLMs demonstrate the usability of SecBench, which is arguably the largest and most comprehensive benchmark dataset for LLMs in cybersecurity. More information about SecBench can be found at our website, and the dataset can be accessed via the artifact link.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑