arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11892cs.SEcs.AI
收藏

面向具有自然鲁棒性测试的神经程序修复可靠评估

Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing

The University of Melbourne(墨尔本大学)

Thanh Le-Cong, Dat Nguyen, Bach Le, Toby Murray

AI总结:

本文提出通过自然鲁棒性测试评估神经程序修复技术,发现仅60%的语义保持转换是自然的,不自然转换显著影响评估结论,并导致补丁率显著下降,最后提出基于LLM的自动化自然性评估指标。

摘要:

本文提出将神经程序修复(NPR)技术的鲁棒性评估重点转向自然发生的数据转换。为此,我们首先通过一项两阶段的人工研究,考察了保持语义转换的自然性。该研究包括:(1)访谈资深软件开发者,为评估这些转换的自然性确立具体标准;(2)对10名开发者进行调查,以评估应用于225个真实世界漏洞的1,178个转换(即原始程序与转换后程序对)的自然性。研究结果表明,仅60%的转换被认为是自然的,20%被认为是不自然的,且标注者之间具有高度一致性。此外,这些转换的不自然性显著影响了它们在基准测试中的适用性以及从鲁棒性测试中得出的结论。接下来,我们对NPR技术进行自然鲁棒性测试,以评估其应对真实世界数据变化的真实有效性。实验结果显示,NPR技术出现了大量预测改变,导致在原始数据集和转换后数据集上的表现对比中,合理补丁率和正确补丁率均显著下降。此外,我们观察到NPR技术之间的性能提升存在显著差异,表明有限的数据集对NPR评估引入了潜在偏差。最后,我们提出了一种基于LLM的指标,以自动化评估转换的自然性,确保自然鲁棒性测试的可扩展性。

英文摘要:

In this paper, we propose shifting the focus of robustness evaluation for Neural Program Repair (NPR) techniques toward naturally-occurring data transformations. To accomplish this, we first examine the naturalness of semantic-preserving transformations through a two-stage human study. This study includes (1) interviews with senior software developers to establish concrete criteria for evaluating the naturalness of these transformations, and (2) a survey involving 10 developers to assess the naturalness of 1,178 transformations, i.e., pairs of original and transformed programs, applied to 225 real-world bugs. Our findings show that only 60% of these transformations are deemed natural, while 20% are considered unnatural, with strong agreement among annotators. Moreover, the unnaturalness of these transformations significantly impacts both their applicability to benchmarks and the conclusions drawn from robustness testing. Next, we conduct natural robustness testing on NPR techniques to assess their true effectiveness against real-world data variations. Our experimental results reveal a substantial number of prediction changes in NPR techniques, leading to significant reductions in both plausible and correct patch rates when comparing performance on the original and transformed datasets. Additionally, we observe notable differences in performance improvements between NPR techniques, suggesting potential biases on NPR evaluation introduced by limited datasets. Finally, we propose an LLM-based metric to automate the assessment of transformation naturalness, ensuring the scalability of natural robustness testing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑