arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.19239cs.SE
收藏

基于大语言模型的项目级漏洞检测:一项实证研究

LLM-based Vulnerability Detection at Project Scale: An Empirical Study

School of Computer Software, College of Intelligence and Computing, Tianjin University(天津大学计算机科学与技术学院);International Joint Institute of Tianjin University(天津大学国际联合学院);Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, School of Computer Science, Peking University(北京大学计算机科学学院,教育部高可信软件技术重点实验室)

Fengjie Li, Jiajun Jiang, Dongchi Chen, Yingfei Xiong

AI总结:

本文研究了基于LLM的项目级漏洞检测方法,发现其在召回率低的情况下仍能发现更多漏洞,但存在高误报率和计算成本问题。

摘要:

在本文中,我们首次对专门用于漏洞检测的大型语言模型(LLM)检测器进行了全面的实证研究,并将其与传统静态分析器在项目层面进行比较。具体而言,我们的研究评估了五个最新且具有代表性的LLM方法和两种传统工具,使用了两种方法:1)一个包含222个已知现实世界漏洞(C/C++和Java)的内部基准,以评估检测能力;2)24个活跃的开源项目,其中我们手动检查了385个警告,以评估其实际可用性和潜在的失败根本原因。我们的评估得出三个关键发现:首先,尽管LLM检测器在内部基准上召回率较低,但它们仍然发现了比传统工具更多的独特漏洞。其次,在开源项目中,LLM和传统工具都生成了大量的警告,但存在非常高的假发现率,这阻碍了实际应用。我们的手动分析进一步揭示了浅层过程推理和错误识别源/汇对作为主要失败原因,LLM工具还表现出额外的唯一失败。最后,LLM方法带来了显著的计算成本——数以万计到数以千万个标记,以及数小时到数天的运行时间。总体而言,我们的发现强调了当前LLM检测器在鲁棒性、可靠性和可扩展性方面的关键限制。我们最终总结了一组对未来研究的启示,以实现更有效和实用的项目级漏洞检测。

英文摘要:

As software complexity grows, automated vulnerability detection becomes increasingly important. Recent LLM-based detectors combine semantic reasoning with static analysis for project-scale scanning, but their practical effectiveness and failure causes remain unclear. We present the first comprehensive empirical study of specialized LLM-based detectors at project scale, evaluating five specialized methods, two general-purpose agents, and four traditional static analyzers on 265 known C/C++ and Java vulnerabilities and 24 active open-source projects/modules. Using Codex-assisted labeling with stratified manual validation, we analyze 6,442 sampled warnings and classify 5,896 false positives using a taxonomy derived from 355 manually inspected reports. Our study yields three findings. First, general-purpose agents achieve the highest recall but struggle with complex code, while both LLM-based and traditional methods suffer from incomplete API modeling. Second, many tools exhibit high false discovery rates on real-world projects/modules. Truncated interprocedural context and incorrect program-point classification are the dominant false-positive causes, with LLM-based tools exhibiting additional reasoning and prompt-compliance failures. Third, LLM-based methods incur substantial costs: hundreds of thousands to hundreds of millions of tokens, API charges exceeding $1,000 per project/module, and runtimes ranging from hours to days. These findings reveal limitations in the robustness, reliability, and scalability of current detectors and inform directions for more effective and practical project-scale vulnerability detection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑