ReCon：用于跨摄取和检索管道的基于大语言模型的网络安全合规性的资源受限基准测试
ReCon: A Resource-Constrained Benchmark for LLM-Based Cybersecurity Compliance Across Ingestion and Retrieval Pipelines
AI总结：
研究政府、企业和机构网络安全合规问题，利用无需GPU和高内存的大语言模型进行合规检查任务基准测试，实验证明低资源大语言模型在政策文件合规检查中可提供良好一致性/准确性。
AI中文摘要：
随着政府、企业和机构面临的网络威胁形势日益严峻，网络/信息安全治理的一个主要组成部分是制定、采用和实施全面的安全政策文件，该文件须符合国际或国家标准及监管指南。但政策文件常不完整，识别其与标准间差距需大量人工。生成式人工智能工具出现后，有人用大语言模型和智能人工智能工具自动化合规检查，但多在高资源环境试验。本文对无需GPU和高内存使用的大语言模型进行合规检查任务基准测试，实验表明低资源大语言模型在针对标准的政策文件合规检查中能提供良好的一致性/准确性。
英文摘要：
With the increasingly aggressive cyber threat landscape for governments, businesses, and institutions, as information and/or cybersecurity implementations are increasingly under scrutiny by regulators, it has been pointed out that governance failure is one of the major reasons for a weakened cybersecurity posture. A major component of Cyber/information security governance is the development, adoption, and implementation of a comprehensive information and/or cyber security policy document. The policy document must be in compliance with international or national standards and, if possible, with regulatory guidelines. However, it is often observed that policy documents are often incomplete with respect to industry standards or regulations and require revision when subjected to a thorough audit. Identifying the gaps between the controls and processes documented in the policy and those required in the regulations or standards necessitates extensive manual effort. The advent of Generative AI tools such as Large Language Models (LLMs) led to use of LLMs and Agentic AI tools to automate such compliance checks, as seen in a few research publications in recent times. However, such reported use of LLMs are experimented with high resource environments such as expensive GPUs and memory based servers. For smaller organizations such expensive compute platform may not be easily available. In this article, we benchmark the compliance checking tasks on LLMs that do not require GPU and high memory usage and the effectiveness of such resource constrained LLMs in compliance checking. Our experiments demonstrated that the low resource LLMs can provide good agreement/accuracy in compliance checking of policy documents against standards by experimenting with ISO 27002:2022 controls against multiple policy documents.
