Automated Program Repair (APR) techniques have shown more and more promising results in fixing real-world bugs. Despite the effectiveness, APR techniques still face an overfitting problem: a generated patch can be incorrect although it passes all tests. It is time-consuming to manually evaluate the correctness of generated patches that can pass all tests. To address this problem, many approaches have been proposed to automatically assess the correctness of patches generated by APR techniques. These approaches are mainly evaluated within the cross-validation setting. However, for patches generated by a new or unseen APR tool, users are implicitly required to manually label a significant portion of these patches in the cross-validation setting before inferring the remaining patches. To mitigate the issue, in this study, we propose \toolname, the patch correctness assessment by adopting a large language model for code. Specifically, for patches generated by a new or unseen APR tool, \toolname does not need labeled patches of this new or unseen APR tool for training but directly queries the large language model for code to get predictions on the correctness labels without training. In this way, \toolname can reduce the manual labeling effort when building a model to automatically assess the correctness of generated patches of new APR tools. \toolname prioritizes labeled patches from existing APR tools that exhibit semantic similarity to those generated by new APR tools, enhancing the accuracy achieved by \toolname for patches from new APR tools. Our experimental results showed that \toolname can achieve an accuracy of 84.4% and an F1-score of 86.5% on average although no labeled patch of the new or unseen APR tool is available. In addition, our proposed technique outperformed the prior state-of-the-art by a large margin.
Attention: Not Just Another Dataset for Patch-Correctness Checking
自动程序修复(Automated Program Repair,APR)技术已引起学术界和工业界的广泛关注。同时,当前最先进的APR工具的一个主要限制是,通过所有原始测试的补丁不一定是开发人员想要的正确补丁,即,合理的补丁问题。迄今为止,已经提出了各种补丁正确性检查(PCC)技术来解决这个重要问题。然而,它们仅在非常有限的数据集上进行评估,因为用于生成此类补丁的APR工具只能探索可能补丁的搜索空间的一小部分,对现有PCC研究的外部有效性构成严重威胁。在本文中,我们构建了一个广泛的PCC数据集(据我们所知最大的手动标记的PCC数据集),以重新审视所有最先进的PCC技术。更具体地说,我们的PCC数据集包括从最近的PraPR APR工具生成的1,988个补丁,该工具利用高度优化的字节码级别补丁执行,并可以在其大型预定义的搜索空间内彻底探索所有可能的合理补丁(包括众所周知的修复模式)来自各种以前的APR工具)。我们对新数据集上代表性PCC技术的广泛研究揭示了各种令人惊讶的发现,并为未来的PCC研究提供了指导。
A Critical Review of "Automatic Patch Generation Learned from Human-Written Patches": Essay on the Problem Statement and the Evaluation of Automatic Software Repair
How Different Is It Between Machine-Generated and Developer-Provided Patches? An Empirical Study on The Correct Patches Generated by Automated Program Repair Techniques
背景资料:多年来,自动程序修复(Automated Program Repair,APR)由于能够降低修复错误的成本而受到学术界和工业界的广泛关注。然而,如何评估补丁的正确性仍然是一个开放的挑战。两种广泛采用的方法来应对这一挑战,包括手动检查和使用自动生成的测试进行验证,都是有偏见的(即,分别遭受主观性和低精度)。目的:为了解决这一问题,我们建议进行实证研究,了解正确的补丁,由现有的国家的最先进的APR技术,旨在为未来的补丁评估提供指导方针。方法:为此,我们首先提出了一个文献综述(LR)报告的正确补丁生成的最新技术的缺陷4J基准和收集177个正确的补丁后,健全检查的过程。我们研究了这些机器生成的正确补丁是如何实现语义等价的,但与开发人员提供的补丁相比在语法上存在差异,这些补丁如何在不同的项目和APR技术中分布,以及错误的特征如何影响为其生成的补丁。我们的主要发现包括:1)我们不需要像开发人员那样修复bug,因为我们观察到25.4%(45/177)2)机器生成的正确补丁的分布在Defects 4J项目和APR技术方面存在差异;以及3)APR技术倾向于生成与开发者针对具有大补丁大小的错误所生成的补丁不同的补丁。结论:我们的研究不仅验证了以前的研究结论,但也强调了未来的研究对评估补丁的正确性的影响。
Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术开发区(滨江)区块链与数据安全研究院);Graz University of Technology(格拉茨工业大学)
该研究以SWE-bench等基准的F2P测试为工具,搭配Claude Opus 5等模型与SWE-agent,发现LLM生成补丁的运行时性能与问题解决率无关,需将补丁性能纳入仓库级问题解决的评估。
Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches
University of Luxembourg(卢森堡大学);Nanjing University of Science and Technology(南京理工大学);Zhejiang University(浙江大学);Bilkent University(比尔肯特大学);Singapore Management University(新加坡管理大学)
When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
Carnegie Mellon University(卡内基梅隆大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);Peking University(北京大学);Google(谷歌);Colorado State University(科罗拉多州立大学)
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
Singapore Management University(新加坡国立管理学院);DGIST(韩国高等智能科学研究院);Monash University(墨尔本大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Huawei Software Engineering Application Technology Lab(华为软件工程应用技术实验室);Zhejiang University(浙江大学)
Automated Exploit and Security Test Generation: A Systematic Literature Review
Hamburg University of Technology(汉堡工业大学);University of Trento(特伦托大学);University of Szeged(塞格德大学);University of Salerno(萨莱诺大学);FrontEndART Ltd.(FrontEndART有限公司);Vrije Universitiet Amsterdam(阿姆斯特丹自由大学)