arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13291cs.CRcs.LGcs.PLcs.SE
收藏

DeepCode AI Fix:利用大型语言模型修复安全漏洞

DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models

Snyk;INSAIT;Sofia University(索非亚大学);ETH Zurich(苏黎世联邦理工学院)

Berkay Berabi, Alexey Gronskiy, Veselin Raychev, Gishor Sivanrupan, Victor Chibotaru, Martin Vechev

AI总结:

本文提出通过程序分析缩减代码片段以聚焦LLM注意力的方法,降低修复安全漏洞所需训练数据量,并构建了涵盖156种缺陷模式的数据集,使Mixtral-8x7B系统可消除超80%缺陷,优于GPT-3.5、GPT-4和TFix等基线。

摘要:

自动化程序修复领域多年来吸引了大量关注,但尽管进行了大量研究工作,创建一个能够有效处理诸如安全漏洞等复杂语义缺陷的系统仍然十分困难。解决这一挑战的一个有前景的方向是利用大型语言模型(LLMs),这些模型正越来越多地被用于解决各种编程任务。在本文中,我们研究了LLMs在解决代码修复任务中的有效性。我们表明该任务是困难的,因为它要求模型学习长距离的代码关系,而这一任务本质上依赖于大量的训练数据。与此同时,为复杂程序缺陷及其对应修复创建大规模、干净的数据集并非易事。我们提出了一种技术,通过一种新的查询和微调LLMs的方法来应对这些挑战。其思想是利用程序分析来限制LLM的注意力机制,使其只关注执行修复所需的代码部分,从而大幅减少所需的训练数据量。具体而言,在训练和推理过程中,我们不是将整个程序输入给LLM,而是将其代码缩减为一个短得多的片段,该片段包含所报告的缺陷以及必要的上下文,并使用该片段代替整个程序。我们的评估表明,这种代码缩减方法显著改善了现有模型(如使用少样本学习的GPT-4)以及微调模型的性能。为了训练和评估我们的系统,我们通过广泛标注156种缺陷模式(包括40条安全规则)创建了一个全面的代码修复数据集,这些缺陷模式的发现需要复杂的跨过程数据流分析。我们使用Mixtral-8x7B的最佳系统能够消除超过80%的报告缺陷,同时在10%到50%的情况下与人类修复完全匹配,优于基于GPT-3.5和GPT-4的基线方法,以及基于窗口的模型如TFix。

英文摘要:

The automated program repair field has attracted substantial interest over the years, but despite significant research efforts, creating a system that works well for complex semantic bugs such as security vulnerabilities has proven difficult. A promising direction to solve this challenge is by leveraging large language models (LLMs), which are increasingly used to solve various programming tasks. In this paper, we investigate the effectiveness of LLMs for solving code-repair task. We show that the task is difficult as it requires the model to learn long-range code relationships, a task that inherently relies on extensive amounts of training data. At the same time, creating a large, clean dataset for complex program bugs and their corresponding fixes is non-trivial. We propose a technique to address these challenges with a new approach for querying and fine-tuning LLMs. The idea is to use program analysis to limit the LLM's attention mechanism on the portions of code needed to perform the fix, drastically reducing the amount of required training data. Concretely, for training and inference, rather than feeding the entire program to the LLM, we reduce its code to a much shorter snippet that contains the reported defect together with the necessary context - and use that instead. Our evaluation shows that this code reduction approach substantially improves available models such as GPT-4 using few-shot learning, as well as fine-tuning models. To train and evaluate our system, we created a comprehensive code fixing dataset by extensively labeling 156 bug patterns (including 40 security rules), requiring complex interprocedural dataflow to discover. Our best system with Mixtral-8x7B can remove more than 80% of the reported defects while exactly matching the human fix in between 10 and 50% of cases, outperforming baselines based on GPT-3.5 and GPT-4, or based on window-based models like TFix.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑