不再需要人工测试?评估并改进 ChatGPT 的单元测试生成
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
AI总结:
本文首次实证评估 ChatGPT 的单元测试质量,并提出含初始生成与迭代精炼的 ChatTESTER,显著提升可编译性和正确断言比例。
摘要:
单元测试对于检测功能离散程序单元中的缺陷至关重要。手动编写高质量单元测试既耗时又费力。尽管传统技术能够生成具有合理覆盖率的测试,但它们可读性较低,且无法被开发者直接采用。近期研究表明,大型语言模型(LLM)在单元测试生成方面具有巨大潜力,能够生成更接近人类、更有意义的测试代码。ChatGPT 是融合指令微调和强化学习的最新 LLM,已在多个领域表现出色。然而,ChatGPT 在单元测试生成方面究竟效果如何仍不清楚。在本工作中,我们开展首个实证研究,以评估 ChatGPT 的单元测试生成能力。具体而言,我们进行定量分析和用户研究,从正确性、充分性、可读性和可用性方面系统考察其生成测试的质量。ChatGPT 生成的测试仍存在正确性问题,包括多种编译错误和执行失败。尽管如此,ChatGPT 生成的通过测试在覆盖率、可读性上可与人工编写的测试相媲美,甚至有时更受开发者偏好,因而与人工测试相似。我们的发现表明,如果能够进一步提升其生成测试的正确性,使用 ChatGPT 生成单元测试将非常有前景。受上述发现启发,我们提出 ChatTESTER,这是一种新颖的基于 ChatGPT 的单元测试生成方法,利用 ChatGPT 自身提升其生成测试的质量。ChatTESTER 包含初始测试生成器和迭代测试精炼器。我们的评估证明了 ChatTESTER 的有效性:与默认 ChatGPT 相比,其生成的可编译测试多 34.3%,具有正确断言的测试多 18.7%。
英文摘要:
Unit testing is essential in detecting bugs in functionally-discrete program units. Manually writing high-quality unit tests is time-consuming and laborious. Although traditional techniques can generate tests with reasonable coverage, they exhibit low readability and cannot be directly adopted by developers. Recent work has shown the large potential of large language models (LLMs) in unit test generation, which can generate more human-like and meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning and reinforcement learning, has performed well in various domains. However, It remains unclear how effective ChatGPT is in unit test generation. In this work, we perform the first empirical study to evaluate ChatGPT's capability of unit test generation. Specifically, we conduct a quantitative analysis and a user study to systematically investigate the quality of its generated tests regarding the correctness, sufficiency, readability, and usability. The tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures. Still, the passing tests generated by ChatGPT resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTESTER incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。