arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.26674cs.SE
收藏

可重复的自动程序修复是困难的——Defects4J数据集的经验

Reproducible Automated Program Repair Is Hard -- Experiences With the Defects4J Dataset

University of Hildesheim(希尔德斯海姆大学)

Adam Krafczyk, Klaus Schmid

AI总结:

本文探讨了Defects4J数据集在自动程序修复中的挑战,发现21.6%的缺陷无法重复,7.1%的测试套件不足,提出了更严格的评估框架以解决测试套件问题。

摘要:

在自动程序修复(APR)研究中,包含已知缺陷和指示缺陷的测试套件的基准数据集非常重要。我们发现使用广泛使用的缺陷数据集时存在显著挑战,超越了单纯通过测试用例重复缺陷。本文总结了这些挑战和经验教训,以引起APR社区的注意,并量化其潜在影响。特别研究了广泛使用的基准数据集Defects4J,包含17个开源Java项目的835个缺陷;手选的缺陷、明确指示缺陷的测试套件和人类补丁,其中移除了无关更改。发现当执行测试套件时,严格要求可重复性(超越单纯通过测试用例重复缺陷),180(21.6%)的缺陷不适合评估实验。此外,发现另外59(7.1%)的缺陷的测试套件明显不足,删除代码库中的单个语句会使所有测试用例通过,尽管人类补丁不仅删除代码。我们的贡献包括:对APR缺陷数据集的系统性要求、对Defects4J数据集的实践经验描述和定量分析,以及为Java程序的APR工具实现评估框架。该评估框架对不足的测试套件进行更严格的检查,以避免测试套件中未被注意到的问题,如不稳定测试。

英文摘要:

In the research of automated program repair (APR), benchmark datasets consisting of known defects in combination with test suites that indicate the defects are of high importance. They allow for an evidence-based comparison of different APR approaches. In our own work on APR we found significant challenges when working with widely used defect datasets, which go beyond mere repeatability of defects via test cases. We summarize these identified challenges and related lessons learned to bring them to the attention of the APR community and quantify the potential impact of them. In particular, we investigate the widely used benchmark Defects4J, which has according to Google Scholar over 1,800 citations. It consists of 835 defects from 17 open-source Java projects; a hand-curated collection of defects, test suites that clearly indicate the defect, and human patches where any unrelated changes are removed. We find that, when executing the test suites with strict requirements for reproducibility in APR settings (beyond merely reproducing the defect via test cases), 180 (21.6 %) of the defects are not suitable for evaluation experiments. Further, we find that an additional 59 (7.1 %) defects have test suites that are obviously under-specified, as deleting a single statement from the code base makes all test cases pass, although the human-written patch does not only delete code. Our contributions are: a systematic collection of requirements for defect datasets for APR beyond traditional reproducibility of defects, a description of practical experiences and quantitative analysis of problems with the Defects4J dataset, as well as an implementation of an evaluation framework for APR tools for Java programs. This evaluation framework does stricter checking for indications of inadequate test suites, to avoid otherwise unnoticed problems in the test suite, such as flaky tests.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑