通过大语言模型进行安全代码审查的洞察：能力、障碍及影响因素
An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors
AI总结：
本文通过实证研究探讨大语言模型在安全代码审查中的潜力，比较了七种LLM与静态分析工具的性能，发现LLM在检测安全缺陷方面表现优异，且特定提示策略对性能有显著影响。
摘要：
安全代码审查是一个耗时且劳动密集的过程，通常需要与自动化安全缺陷检测工具集成。然而，现有安全分析工具在泛化能力、假阳性率和检测粒度方面表现不佳。大语言模型（LLMs）被认为能够解决这些挑战。在本研究中，我们进行了实证研究，探讨LLM在代码审查中检测安全缺陷的潜力。具体而言，我们评估了七种LLM在五个不同提示下的性能，并将其与最先进的静态分析工具进行比较。我们还对表现最佳的两种LLM进行了语言和回归分析，以识别其响应中的质量问题及影响性能的因素。我们的发现表明：（1）在安全代码审查中，LLM显著优于最先进的静态分析工具，且推理优化的LLM比通用LLM表现更好。（2）DeepSeek-R1取得了最高性能，其次是ChatGPT平台提供的GPT-4。DeepSeek-R1的最佳提示结合了提交信息和推理链（CoT）指导，而GPT-4 via ChatGPT的最佳提示使用了常见弱点枚举（CWE）列表。（3）GPT-4 via ChatGPT频繁产生模糊表达，难以准确遵循提示指令，而DeepSeek-R1更常生成不准确的代码细节。（4）LLM在识别具有较少token和安全相关注释的代码文件中的安全缺陷方面更擅长。（5）代码复杂性与DeepSeek-R1对特定安全缺陷类型的检测能力呈正相关。
英文摘要：
Security code review is a time-consuming and labor-intensive process typically requiring integration with automated security defect detection tools. However, existing security analysis tools struggle with poor generalization, high false positive rates, and coarse detection granularity. Large Language Models (LLMs) have been considered promising candidates for addressing those challenges. In this study, we conducted an empirical study to explore the potential of LLMs in detecting security defects during code review. Specifically, we evaluated the performance of seven LLMs under five different prompts and compared them with state-of-the-art static analysis tools. We also performed linguistic and regression analyses for the two top-performing LLMs to identify quality problems in their responses and factors influencing their performance. Our findings show that: (1) In security code review, LLMs significantly outperform state-of-the-art static analysis tools, and the reasoning-optimized LLM performs better than general-purpose LLMs. (2) DeepSeek-R1 achieves the highest performance, followed by GPT-4 provided in the ChatGPT platform. The optimal prompt for DeepSeek-R1 incorporates both the commit message and chain-of-thought (CoT) guidance, while for GPT-4 via ChatGPT, the prompt with a Common Weakness Enumeration (CWE) list works best. (3) GPT-4 via ChatGPT frequently produces vague expressions and exhibits difficulties in accurately following instructions in the prompts, while DeepSeek-R1 more commonly generates inaccurate code details in its outputs. (4) LLMs are more adept at identifying security defects in code files that have fewer tokens and security-relevant annotations. (5) Higher code complexity correlates with enhanced detection capabilities of DeepSeek-R1 for specific security defect types.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
