arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.03470cs.SE
收藏

大语言模型在多语言漏洞修复中的评估研究

On the Evaluation of Large Language Models in Multilingual Vulnerability Repair

Dong wang, Junji Yu, Honglin Shu, Michael Fu, Chakkrit Tantithamthavorn, Yasutaka Kamei, Junjie Chen

AI总结:

本研究通过七种编程语言的大规模实证,评估了LLM在多语言漏洞修复中的性能,发现指令微调的GPT-4o表现优异,尤其擅长修复独特和危险漏洞,并展现了跨语言泛化能力。

摘要:

已有多种基于深度学习的方法利用预训练语言模型来自动修复软件漏洞。然而,这些方法仅限于特定的编程语言(C/C++)。大语言模型(LLM)的最新进展提供了语言无关的能力和强大的语义理解,展现出克服多语言漏洞修复局限的潜力。尽管已有一些工作开始探索LLM的修复性能,但其效果仍不尽人意。为解决这些局限,我们开展了一项大规模实证研究,调查了自动化漏洞修复方法和最先进的LLM在七种编程语言上的性能。结果表明,经过指令微调并采用少样本提示的GPT-4o,其表现与领先方法VulMaster不相上下。此外,基于LLM的方法在修复独特漏洞方面表现出更优的性能,并且更有可能修复最危险的漏洞。指令微调的GPT-4o在修复先前未见语言中的漏洞时展现出强大的泛化能力,优于现有方法。分析显示,Go在所有模型类型中始终取得最高修复效果,而C/C++表现最差。基于这些发现,我们讨论了LLM在多语言漏洞修复中的前景以及LLM失败案例背后的原因。这项工作首次对多种语言下的修复方法和LLM进行了考察,凸显了采用LLM进行多语言漏洞修复的光明未来。

英文摘要:

Various Deep Learning-based approaches with pre-trained language models have been proposed for automatically repairing software vulnerabilities. However, these approaches are limited to a specific programming language (C/C++). Recent advances in large language models (LLMs) offer language-agnostic capabilities and strong semantic understanding, exhibiting potential to overcome multilingual vulnerability limitations. Although some work has begun to explore LLMs' repair performance, their effectiveness is unsatisfactory. To address these limitations, we conducted a large-scale empirical study to investigate the performance of automated vulnerability repair approaches and state-of-the-art LLMs across seven programming languages. Results show GPT-4o, instruction-tuned with few-shot prompting, performs competitively against the leading approach, VulMaster. Additionally, the LLM-based approach shows superior performance in repairing unique vulnerabilities and is more likely to repair the most dangerous vulnerabilities. Instruction-tuned GPT-4o demonstrates strong generalization on vulnerabilities in previously unseen language, outperforming existing approaches. Analysis shows Go consistently achieves the highest effectiveness across all model types, while C/C++ performs the worst. Based on findings, we discuss the promise of LLM on multilingual vulnerability repair and the reasons behind LLM's failed cases. This work takes the first look at repair approaches and LLMs across multiple languages, highlighting the promising future of adopting LLMs for multilingual vulnerability repair.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑