arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2408.03095cs.SE
收藏

TestART:通过自动生成与修复迭代的协同进化改进基于LLM的单元测试

TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration

Siqi Gu, Quanjun Zhang, Kecheng Li, Chunrong Fang, Fangyuan Tian, Liuchuan Zhu, Jianyi Zhou, Zhenyu Chen

AI总结:

TestART通过生成与修复的协同进化,利用模板修复、覆盖率反馈和正向提示注入,提升LLM单元测试的通过率与覆盖率。

摘要:

单元测试对于检测单个程序单元中的缺陷至关重要,但会消耗时间和精力。近年来,大型语言模型(LLMs)在生成单元测试用例方面展示了卓越的能力。然而,若干问题限制了它们生成高质量单元测试用例的能力:(1)由LLMs的幻觉引起的编译和运行时错误;(2)缺乏测试和覆盖率反馈信息,限制了代码覆盖率的提高;(3)重复抑制问题导致基于LLM的修复和生成尝试无效。为了解决这些局限性,我们提出了TestART,一种新颖的单元测试生成方法。TestART通过自动生成和修复迭代的协同进化来改进基于LLM的单元测试,代表了自动单元测试生成的重大进步。TestART首次利用基于模板的修复策略来有效修复LLM生成的测试用例中的错误。同时,TestART从成功的测试用例中提取覆盖率信息,并将其用作覆盖率引导的测试反馈。它还结合了正向提示注入以防止重复抑制,从而增强了最终测试用例的充分性。这种生成与修复之间的协同作用使生成的测试用例的正确性和充分性显著超越先前的方法。在对比实验中,与基线模型相比,TestART在三种类型的数据集上实现了通过率提高18%和覆盖率提升20%。此外,与EvoSuite相比,它仅用一半数量的测试用例就实现了更好的覆盖率。这些结果表明,TestART通过利用LLMs的能力同时克服其固有缺陷,具有生成高质量单元测试用例的卓越能力。

英文摘要:

Unit testing is crucial for detecting bugs in individual program units but consumes time and effort. Recently, large language models (LLMs) have demonstrated remarkable capabilities in generating unit test cases. However, several problems limit their ability to generate high-quality unit test cases: (1) compilation and runtime errors caused by the hallucination of LLMs; (2) lack of testing and coverage feedback information restricting the increase of code coverage;(3) the repetitive suppression problem causing invalid LLM-based repair and generation attempts. To address these limitations, we propose TestART, a novel unit test generation method. TestART improves LLM-based unit testing via co-evolution of automated generation and repair iteration, representing a significant advancement in automated unit test generation. TestART leverages the template-based repair strategy to effectively fix bugs in LLM-generated test cases for the first time. Meanwhile, TestART extracts coverage information from successful test cases and uses it as coverage-guided testing feedback. It also incorporates positive prompt injection to prevent repetition suppression, thereby enhancing the sufficiency of the final test case. This synergy between generation and repair elevates the correctness and sufficiency of the produced test cases significantly beyond previous methods. In comparative experiments, TestART demonstrates an 18% improvement in pass rate and a 20% enhancement in coverage across three types of datasets compared to baseline models. Additionally, it achieves better coverage rates than EvoSuite with only half the number of test cases. These results demonstrate TestART's superior ability to produce high-quality unit test cases by harnessing the power of LLMs while overcoming their inherent flaws.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑