arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.00202cs.SE
收藏

PatchZero:零样本自动补丁正确性评估

PatchZero: Zero-Shot Automatic Patch Correctness Assessment

Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Thanh Le-Cong, Junda He, Bach Le, David Lo

AI总结:

针对自动程序修复中补丁正确性评估需大量人工标注的问题,提出基于代码大模型的零样本评估方法PatchZero,无需新工具的标注数据即可预测,准确率达84.4%并大幅超越现有技术。

摘要:

自动程序修复(APR)技术在修复真实世界的错误方面展现出了越来越有前景的结果。尽管十分有效,APR技术仍然面临过拟合问题:生成的补丁可能通过所有测试,但却是错误的。手动评估那些通过所有测试的生成补丁的正确性非常耗时。为了解决这个问题,已经提出了许多方法来自动评估APR技术生成的补丁的正确性。这些方法主要在交叉验证设置下进行评估。然而,对于由新的或未见过的APR工具生成的补丁,用户在交叉验证设置下,在推断剩余补丁之前,被隐式要求手动标注这些补丁的很大一部分。为了缓解这个问题,在本研究中,我们提出了PatchZero,通过采用代码大模型来进行补丁正确性评估。具体来说,对于由新的或未见过的APR工具生成的补丁,PatchZero不需要该新或未见APR工具的已标注补丁进行训练,而是直接查询代码大模型以获取正确性标签的预测,无需训练。这样,PatchZero可以减少在构建模型以自动评估新APR工具生成补丁的正确性时的人工标注工作。PatchZero优先选择来自现有APR工具的、与由新APR工具生成的补丁表现出语义相似性的已标注补丁,从而提高PatchZero对来自新APR工具补丁的准确性。我们的实验结果表明,尽管没有新或未见APR工具的已标注补丁可用,PatchZero平均可以达到84.4%的准确率和86.5%的F1分数。此外,我们提出的技术大幅超越了之前的最先进技术。

英文摘要:

Automated Program Repair (APR) techniques have shown more and more promising results in fixing real-world bugs. Despite the effectiveness, APR techniques still face an overfitting problem: a generated patch can be incorrect although it passes all tests. It is time-consuming to manually evaluate the correctness of generated patches that can pass all tests. To address this problem, many approaches have been proposed to automatically assess the correctness of patches generated by APR techniques. These approaches are mainly evaluated within the cross-validation setting. However, for patches generated by a new or unseen APR tool, users are implicitly required to manually label a significant portion of these patches in the cross-validation setting before inferring the remaining patches. To mitigate the issue, in this study, we propose \toolname, the patch correctness assessment by adopting a large language model for code. Specifically, for patches generated by a new or unseen APR tool, \toolname does not need labeled patches of this new or unseen APR tool for training but directly queries the large language model for code to get predictions on the correctness labels without training. In this way, \toolname can reduce the manual labeling effort when building a model to automatically assess the correctness of generated patches of new APR tools. \toolname prioritizes labeled patches from existing APR tools that exhibit semantic similarity to those generated by new APR tools, enhancing the accuracy achieved by \toolname for patches from new APR tools. Our experimental results showed that \toolname can achieve an accuracy of 84.4% and an F1-score of 86.5% on average although no labeled patch of the new or unseen APR tool is available. In addition, our proposed technique outperformed the prior state-of-the-art by a large margin.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑