SECURE：面向网络安全的大语言模型基准测试
SECURE: Benchmarking Large Language Models for Cybersecurity
AI总结：
针对现有LLM基准未充分覆盖网络安全实际应用的问题，提出聚焦工业控制领域的SECURE基准，含6个数据集评估3类能力，测试7个主流模型并给出优化建议。
摘要：
大语言模型（LLMs）已在网络安全应用中展现出潜力，但也因幻觉、缺乏真实性等问题导致可信度较低。现有基准提供通用评估，但未充分覆盖LLM在网络安全特定任务中实际应用层面的表现。为填补这一空白，我们提出SECURE（安全提取、理解与推理评估），这是一款用于评估LLM在真实网络安全场景中表现的基准。SECURE包含6个聚焦工业控制系统领域的数据集，基于行业标准来源评估知识提取、理解与推理能力。我们的研究在这些任务上评估了7个最先进的模型，揭示了它们在网络安全场景下的优缺点，并为提升LLM作为网络咨询工具的可靠性提供了建议。
英文摘要：
Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness. Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks. To address this gap, we introduce the SECURE (Security Extraction, Understanding \& Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios. SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources. Our study evaluates seven state-of-the-art models on these tasks, providing insights into their strengths and weaknesses in cybersecurity contexts, and offer recommendations for improving LLMs reliability as cyber advisory tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
