使用生成式AI生成单元测试：自动生成工具的对比性能分析
Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools
AI总结：
本研究对比了ChatGPT与Pynguin在Python单元测试生成上的表现，发现ChatGPT覆盖率相当但断言错误率较高，两者结合并应用提示工程可显著提升测试生成性能。
摘要：
生成单元测试是软件开发中的一项关键任务，需要程序员投入大量时间和精力。大型语言模型（LLMs）的出现为单元测试脚本生成引入了新途径。本研究旨在通过实验探究LLMs（以ChatGPT为具体示例）为Python程序生成单元测试脚本的有效性，并将生成的测试用例与现有单元测试生成器（Pynguin）生成的用例进行比较。在实验中，我们考虑了三种类型的代码单元：1）过程式脚本，2）基于函数的模块化代码，3）基于类的代码。生成的测试用例基于覆盖率、正确性和可读性等标准进行评估。结果显示，ChatGPT在覆盖率方面与Pynguin相当，尽管在某些情况下其表现优于Pynguin。我们还发现，ChatGPT为某些类别生成的断言中约有三分之一是不正确的。结果还表明，ChatGPT和Pynguin在遗漏语句上的重叠极少，因此，结合这两种工具可能会提升单元测试生成的性能。最后，在我们的实验中，提示工程改善了ChatGPT的表现，实现了更高的覆盖率。
英文摘要：
Generating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to experimentally investigate the effectiveness of LLMs, specifically exemplified by ChatGPT, for generating unit test scripts for Python programs, and how the generated test cases compare with those generated by an existing unit test generator (Pynguin). For experiments, we consider three types of code units: 1) Procedural scripts, 2) Function-based modular code, and 3) Class-based code. The generated test cases are evaluated based on criteria such as coverage, correctness, and readability. Our results show that ChatGPT's performance is comparable with Pynguin in terms of coverage, though for some cases its performance is superior to Pynguin. We also find that about a third of assertions generated by ChatGPT for some categories were incorrect. Our results also show that there is minimal overlap in missed statements between ChatGPT and Pynguin, thus, suggesting that a combination of both tools may enhance unit test generation performance. Finally, in our experiments, prompt engineering improved ChatGPT's performance, achieving a much higher coverage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
