arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.08429cs.SEcs.AI
收藏

使用LLM的软件漏洞与功能评估

Software Vulnerability and Functionality Assessment using LLMs

JP Morgan AI Research(摩根大通AI研究院)

Rasmus Ingemann Tuffveson Jensen, Vali Tawosi, Salwa Alamir

AI总结:

本研究探讨了大型语言模型在代码审查中的应用,重点评估其在标记安全漏洞和验证软件功能方面的性能,发现专有模型显著优于开源模型,且部分生成的漏洞描述与真实CWE漏洞相关。

摘要:

虽然代码审查是软件开发过程的核心,但执行起来可能既繁琐又昂贵。在本文中,我们研究了大型语言模型(LLMs)是否以及如何帮助代码审查。我们的研究聚焦于我们认为对良好审查至关重要的两个任务:(i)标记具有安全漏洞的代码,以及(ii)执行软件功能验证,即确保代码满足其预期功能。为了测试这两个任务的性能,我们使用零样本和思维链提示来获得最终的“批准或拒绝”建议。作为数据,我们使用了开创性的代码生成数据集(HumanEval和MBPP)以及来自常见弱点枚举(CWE)的专家编写的具有安全漏洞的代码片段。我们的实验考虑了来自OpenAI的三个专有模型和较小的开源LLM的混合。我们发现前者以较大优势胜过后者。受到有希望的结果的激励,我们最后要求我们的模型提供安全漏洞的详细描述。结果表明,36.7%的LLM生成的描述可以与真实的CWE漏洞相关联。

英文摘要:

While code review is central to the software development process, it can be tedious and expensive to carry out. In this paper, we investigate whether and how Large Language Models (LLMs) can aid with code reviews. Our investigation focuses on two tasks that we argue are fundamental to good reviews: (i) flagging code with security vulnerabilities and (ii) performing software functionality validation, i.e., ensuring that code meets its intended functionality. To test performance on both tasks, we use zero-shot and chain-of-thought prompting to obtain final ``approve or reject'' recommendations. As data, we employ seminal code generation datasets (HumanEval and MBPP) along with expert-written code snippets with security vulnerabilities from the Common Weakness Enumeration (CWE). Our experiments consider a mixture of three proprietary models from OpenAI and smaller open-source LLMs. We find that the former outperforms the latter by a large margin. Motivated by promising results, we finally ask our models to provide detailed descriptions of security vulnerabilities. Results show that 36.7% of LLM-generated descriptions can be associated with true CWE vulnerabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑