arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.04448cs.SE
收藏

大型语言模型与静态代码分析工具:漏洞检测的系统性基准

Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection

Damian Gnieciak, Tomasz Szandala

AI总结:

本研究通过系统基准比较三种静态分析工具与三种大型语言模型在漏洞检测上的性能,发现LLMs在召回率上优势明显,但误报和定位不准,推荐混合流水线使用。

摘要:

现代软件依赖众多自动化测试和质量保证工具来预防错误、缺陷和潜在漏洞。本研究旨在对六种自动化方法进行正面、定量和定性的评估:三种行业标准的基于规则的静态代码分析工具(SonarQube、CodeQL和Snyk Code)以及三种托管在GitHub Models平台上的最先进大型语言模型(GPT-4.1、Mistral Large和DeepSeek V3)。使用一个精选的包含十个真实世界C#项目的套件,其中嵌入了63个常见类别的漏洞,如SQL注入、硬编码密钥和过时依赖,我们测量了经典检测准确性(精确率、召回率、F分数)、分析延迟以及开发人员审查真阳性所需的工作量。基于语言的扫描器取得了更高的平均F-1分数,分别为0.797、0.753和0.750,而静态对应工具分别得分0.260、0.386和0.546。LLMs的优势源于更高的召回率,证实了其在更广泛的代码上下文中进行推理的能力。然而,这一优势伴随着显著的权衡:DeepSeek V3表现出最高的误报率,所有语言模型由于分词伪影而在行或列粒度上错误定位问题。总体而言,语言模型在发现真实漏洞方面成功与传统静态分析器匹敌。然而,其更嘈杂的输出和不精确的定位限制了它们在安全关键审计中的独立使用。因此,我们推荐一种混合流水线:在开发早期使用语言模型进行广泛的、上下文感知的分类,同时保留确定性基于规则的扫描器用于高保证验证。本文发布的开放基准和基于JSON的结果框架为下一代自动化代码安全的可复现、从业者为中心的研究奠定了基础。

英文摘要:

Modern software relies on a multitude of automated testing and quality assurance tools to prevent errors, bugs and potential vulnerabilities. This study sets out to provide a head-to-head, quantitative and qualitative evaluation of six automated approaches: three industry-standard rule-based static code-analysis tools (SonarQube, CodeQL and Snyk Code) and three state-of-the-art large language models hosted on the GitHub Models platform (GPT-4.1, Mistral Large and DeepSeek V3). Using a curated suite of ten real-world C# projects that embed 63 vulnerabilities across common categories such as SQL injection, hard-coded secrets and outdated dependencies, we measure classical detection accuracy (precision, recall, F-score), analysis latency, and the developer effort required to vet true positives. The language-based scanners achieve higher mean F-1 scores,0.797, 0.753 and 0.750, than their static counterparts, which score 0.260, 0.386 and 0.546, respectively. LLMs' advantage originates from superior recall, confirming an ability to reason across broader code contexts. However, this benefit comes with substantial trade-offs: DeepSeek V3 exhibits the highest false-positive ratio, all language models mislocate issues at line-or-column granularity due to tokenisation artefacts. Overall, language models successfully rival traditional static analysers in finding real vulnerabilities. Still, their noisier output and imprecise localisation limit their standalone use in safety-critical audits. We therefore recommend a hybrid pipeline: employ language models early in development for broad, context-aware triage, while reserving deterministic rule-based scanners for high-assurance verification. The open benchmark and JSON-based result harness released with this paper lay a foundation for reproducible, practitioner-centric research into next-generation automated code security.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑