In the research of automated program repair (APR), benchmark datasets consisting of known defects in combination with test suites that indicate the defects are of high importance. They allow for an evidence-based comparison of different APR approaches. In our own work on APR we found significant challenges when working with widely used defect datasets, which go beyond mere repeatability of defects via test cases. We summarize these identified challenges and related lessons learned to bring them to the attention of the APR community and quantify the potential impact of them.
In particular, we investigate the widely used benchmark Defects4J, which has according to Google Scholar over 1,800 citations. It consists of 835 defects from 17 open-source Java projects; a hand-curated collection of defects, test suites that clearly indicate the defect, and human patches where any unrelated changes are removed. We find that, when executing the test suites with strict requirements for reproducibility in APR settings (beyond merely reproducing the defect via test cases), 180 (21.6 %) of the defects are not suitable for evaluation experiments. Further, we find that an additional 59 (7.1 %) defects have test suites that are obviously under-specified, as deleting a single statement from the code base makes all test cases pass, although the human-written patch does not only delete code.
Our contributions are: a systematic collection of requirements for defect datasets for APR beyond traditional reproducibility of defects, a description of practical experiences and quantitative analysis of problems with the Defects4J dataset, as well as an implementation of an evaluation framework for APR tools for Java programs. This evaluation framework does stricter checking for indications of inadequate test suites, to avoid otherwise unnoticed problems in the test suite, such as flaky tests.
Multi-Location Program Repair Strategies Learned from Past Successful Experience
自动程序修复(Automated program repair,APR)在减少软件维护工作量和时间消耗方面具有巨大的潜力,近年来成为软件工程的研究热点,提出了许多方法。多位置程序修复由于其逻辑和结构的复杂性一直是该领域的一个挑战。虽然一些方法并不声称具有解决多位置错误的功能,但它们在实践中为这些缺陷生成正确的补丁。在本文中,我们首先对Defects4J中的多位置错误进行了观察,并将其分为两类(即,相似的和相关的多位置错误)基于其补丁中的修复动作。然后,我们总结了当前工具修复Defects4J中多位置错误的情况。我们分析了当前工具生成的22个补丁,并提出了两个可行的策略来修复多位置错误,通过两个详细的案例研究来说明它们。最后,通过修复两个从未修复过的漏洞,验证了该方法的可行性。通过借鉴以往的成功经验,指出了多位置程序修复的可能途径。
George Mason University(乔治梅森大学);Bangladesh Election Commission(孟加拉国选举委员会);University of Alabama(阿拉巴马大学);Ministry of Finance(财政部);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair
University of Chinese Academy of Sciences(中国科学院大学);State Key Laboratory of Intelligent Game, Institute of Software, CAS(智能游戏国家重点实验室,软件研究所,中国科学院);Beihang University(北京航空航天大学);York University(约克大学)
A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications
School of Artificial Intelligence (School of Software), Yanshan University(燕山大学人工智能学院(软件学院));School of Computer Science, China University of Mining and Technology(中国矿业大学计算机学院);School of Computing and Information Systems, University of Melbourne(墨尔本大学计算与信息系统的学院);Department of Computer Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校计算机科学系);SnT, University of Luxembourg(卢森堡大学科学与技术研究院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);Department of Computer Science, Aalto University(阿尔托大学计算机科学系)
Productive Reproducible Workflows for DNNs: A Case Study for Industrial Defect Detection
University of Glasgow(格拉斯哥大学)
随着深度神经网络(DNN)成为越来越普遍的工作负载,可用于帮助其开发和部署的库和工具的范围显着增长。可扩展的生产质量工具在许可证下免费提供,并且可以访问,即使是小团队也可以非常高效。然而,在研究界,对上述工具的认识和使用并不一定广泛,研究人员可能会错过利用最新工具和工作流程带来的潜在生产力收益。本文介绍了一个案例研究,我们讨论了我们最近的经验,生产一个端到端的人工智能应用程序的工业缺陷检测。我们详细介绍了高级深度学习库、容器化工作流、持续集成/部署管道和开源代码模板,我们利用它们来产生有竞争力的结果,使其他排名解决方案的性能与我们的三个目标数据集相匹配。我们强调了利用这些系统可以带来的价值,即使是研究,并详细介绍了我们的解决方案,并在服务器级GPU上的准确性和推理时间方面展示了我们的最佳结果,以及服务器级CPU和Raspberry Pi 4上的推理时间。