arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.09433cs.CRcs.AIcs.SE
收藏

CASTLE:面向CWE检测的静态代码分析器与LLM基准测试数据集

CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection

Eötvös Loránd University (ELTE)(厄特沃什·罗兰大学(ELTE));University of Oslo(奥斯陆大学);Cyentific AS;Guelma University(盖勒马大学);The University of Manchester(曼彻斯特大学);Federal University of Amazonas(亚马逊联邦大学);Technology Innovation Institute (TII)(技术创新研究院(TII))

Richard A. Dubniczky, Krisztofer Zoltán Horvát, Tamás Bisztray, Mohamed Amine Ferrag, Lucas C. Cordeiro, Norbert Tihanyi

AI总结:

本文提出CASTLE基准框架,评估静态分析工具、LLM和形式化验证工具在CWE漏洞检测中的能力,发现LLM在小代码片段表现优异但随代码规模增长准确性下降,静态分析器误报率高,形式化验证工具在特定漏洞类型上受限。

摘要:

识别源代码中的漏洞至关重要,尤其是在关键软件组件中。现有的方法如静态分析、动态分析、形式化验证以及最近的大语言模型被广泛用于检测安全缺陷。本文介绍了CASTLE(CWE自动化安全测试与底层评估),一个用于评估不同方法漏洞检测能力的基准测试框架。我们使用包含25个常见CWE的250个微基准程序的人工制作数据集,评估了13个静态分析工具、10个LLM和2个形式化验证工具。我们提出了CASTLE Score,一种确保公平比较的新型评估指标。结果显示了关键差异:ESBMC(一种形式化验证工具)最小化了误报,但在超出模型检查范围的漏洞(如弱密码学或SQL注入)上表现不佳。静态分析器存在高误报,增加了开发者的手动验证工作量。LLM在识别小代码片段中的漏洞时,在CASTLE数据集上表现异常出色。然而,随着代码规模增长,其准确性下降,幻觉增加。这些结果表明,LLM可能在未来的安全解决方案中发挥关键作用,特别是在代码补全框架中,它们可以提供实时指导以防止漏洞。数据集可在https://github.com/CASTLE-Benchmark获取。

英文摘要:

Identifying vulnerabilities in source code is crucial, especially in critical software components. Existing methods such as static analysis, dynamic analysis, formal verification, and recently Large Language Models are widely used to detect security flaws. This paper introduces CASTLE (CWE Automated Security Testing and Low-Level Evaluation), a benchmarking framework for evaluating the vulnerability detection capabilities of different methods. We assess 13 static analysis tools, 10 LLMs, and 2 formal verification tools using a hand-crafted dataset of 250 micro-benchmark programs covering 25 common CWEs. We propose the CASTLE Score, a novel evaluation metric to ensure fair comparison. Our results reveal key differences: ESBMC (a formal verification tool) minimizes false positives but struggles with vulnerabilities beyond model checking, such as weak cryptography or SQL injection. Static analyzers suffer from high false positives, increasing manual validation efforts for developers. LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets. However, their accuracy declines, and hallucinations increase as the code size grows. These results suggest that LLMs could play a pivotal role in future security solutions, particularly within code completion frameworks, where they can provide real-time guidance to prevent vulnerabilities. The dataset is accessible at https://github.com/CASTLE-Benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑