筛选噪声:LLM代理在漏洞假阳性过滤中的比较研究
Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering
AI总结:
本文比较了三种先进的LLM代理框架在漏洞假阳性过滤中的性能,展示了LLM代理在减少SAST噪声方面的有效性,但指出其效果依赖于模型和漏洞类型,且存在计算成本差异。
摘要:
静态应用安全测试(SAST)工具对于识别软件漏洞至关重要,但它们通常会产生大量假阳性(FPs),给开发者带来显著的手动核查负担。近年来,大型语言模型(LLM)代理的进步为通过迭代推理、工具使用和环境交互来细化SAST警报提供了有希望的方向。然而,不同LLM代理架构在假阳性过滤中的相对有效性仍不明确。本文介绍了三种最先进的LLM代理框架(即Aider、OpenHands和SWE-agent)在漏洞假阳性过滤中的比较研究。我们使用OWASP基准中的漏洞和现实世界中的开源Java项目评估这些框架。实验结果表明,LLM代理可以去除大多数SAST噪声,将OWASP基准的初始假阳性检测率从超过92%降低到最佳配置下的6.3%。在现实世界的数据集中,最佳配置的LLM代理可以达到93.3%的假阳性识别率,涉及CodeQL警报。然而,代理的优势强烈依赖于backbone和CWE:代理框架在更强的模型如Claude Sonnet 4和GPT-5上显著优于普通提示,但在较弱的backbone上则收益有限或不一致。此外,激进的假阳性减少可能会以压制真实漏洞为代价,突显了重要的权衡。最后,我们观察到不同代理框架在计算成本上存在显著差异。总体而言,我们的研究表明,LLM代理是强大的但不均匀的SAST假阳性过滤解决方案,其实际部署需要仔细考虑代理设计、backbone模型选择、漏洞类别和运营成本。
英文摘要:
In this paper, we present a comparative study of three state-of-the-art LLM-based agent frameworks, i.e., Aider, OpenHands, and SWE-agent, for vulnerability FP filtering. We evaluate these frameworks using the vulnerabilities from the OWASP Benchmark and real-world open-source Java projects. We further conduct a focused post-cutoff C/C++ study using the strongest configuration to test contamination-free generalization and isolate key agentic capabilities. The experimental results show that LLM-based agents can remove the majority of SAST noise, reducing an initial FP detection rate of over 92% on the OWASP Benchmark to as low as 6.3% in the best configuration. On a real-world Java dataset, the best configuration of LLM-based agents can achieve an FP identification rate of up to 93.3% involving CodeQL alerts. However, the benefits of agents are strongly backbone- and CWE-dependent: agentic frameworks significantly outperform vanilla prompting for stronger models such as Claude Sonnet 4 and GPT-5, but yield limited or inconsistent gains for weaker backbones. On the post-cutoff OSS-Fuzz dataset, SWE-agent with Claude Sonnet 4 identifies 95.5% of FPs while maintaining 95.5% precision, compared with a 36.4% FP identification rate for vanilla prompting. Moreover, aggressive FP reduction can come at the cost of suppressing true vulnerabilities, highlighting important trade-offs. Finally, we observe large disparities in computational cost across agent frameworks. Overall, our study demonstrates that LLM-based agents are a powerful but non-uniform solution for SAST FP filtering, and that their practical deployment requires careful consideration of agent design, backbone model choice, vulnerability category, and operational cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。