arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.16655cs.SEcs.AI
收藏

面向程序修复的内存高效大语言模型与语义引导补丁生成

Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generation

The University of Melbourne（墨尔本大学）

Thanh Le-Cong, Bach Le, Toby Murray

备注：
Accepted to ICSE 2026, Research Track
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出FLAMES，一种语义引导补丁生成的大语言模型程序修复技术，通过贪婪解码和最佳优先搜索降低内存消耗83%，并在Defects4J上多修复10个错误。

摘要：

在本文中，我们首先证明，即使对于小型大语言模型（1B-7B参数），增加波束大小也需要大量的GPU使用，导致基于大语言模型的自动程序修复（APR）中高达80%的重复崩溃是由于内存过载引起的。看似简单的减少内存消耗的解决方案是：（1）对大语言模型进行量化，即将大语言模型的权重从高精度值转换为低精度值；（2）使波束搜索顺序化，即依次将每个波束通过模型前向传播，然后将它们拼接回单个输出。然而，我们通过理论分析和实验表明，这些方法仍然不起作用。为了解决这个问题，我们引入了FLAMES，一种新颖的基于大语言模型的APR技术，它采用语义引导的补丁生成来提高修复效果和内存效率。与依赖波束搜索的传统方法不同，FLAMES利用贪婪解码来提高内存效率，同时通过语义引导的最佳优先搜索算法将搜索引导向更多潜在的良好修复候选。在每个解码步骤中，FLAMES使用来自测试验证的语义反馈（例如通过和失败的测试用例数量）来选择最有希望进一步探索的标记。我们在Defects4J上的实证评估表明，与基于大语言模型的APR相比，FLAMES在时间效率不受影响的情况下，内存消耗大幅降低了高达83%。此外，FLAMES在Defects4J上正确修复了133个错误，比最佳基线多修复了10个错误。此外，这些改进也推广到了HumanEval-Java和TransformedD4J数据集，FLAMES分别比最佳基线多生成了12%和36.5%的正确补丁。

英文摘要：

In this paper, we first show that increases in beam size, even for small-sized LLMs (1B-7B params), require extensive GPU usage, leading to up to 80% of recurring crashes due to memory overloads in LLM-based APR. Seemingly simple solutions to reduce memory consumption are (1) to quantize LLM models, i.e., converting the weights of an LLM from high-precision values to lower-precision ones, and (2) to make beam search sequential, i.e., forwarding each beam through the model sequentially and then concatenating them back into a single output. However, we show that these approaches still do not work via both theoretical analysis and experiments. To address this, we introduce FLAMES, a novel LLM-based APR technique that employs semantic-guided patch generation to enhance repair effectiveness and memory efficiency. Unlike conventional methods that rely on beam search, FLAMES utilizes greedy decoding to enhance memory efficiency while steering the search towards more potentially good repair candidates via a semantic-guided best-first search algorithm. At each decoding step, FLAMES uses semantic feedback from test validation, such as the number of passing and failing test cases, to select the most promising token to explore further. Our empirical evaluation on Defects4J shows thatFLAMES substantially reduces memory consumption by up to 83% compared to LLM-based APR without compromising time efficiency. Moreover, FLAMES correctly fixes 133 bugs on Defects4J, fixing 10 bugs more than the best baseline. Additionally, these improvements also generalize to the HumanEval-Java and TransformedD4J datasets, where FLAMES generates 12% and 36.5% more correct patches, respectively, than the best baseline.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑