arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.00418cs.SEcs.LG
收藏

使用大型语言模型生成JUnit测试:一项实证研究

Using Large Language Models to Generate JUnit Tests: An Empirical Study

University of Notre Dame(圣母大学);Pennsylvania State University(宾夕法尼亚州立大学);IQVIA Inc.(艾昆纬公司);United International University(联合国际大学)

Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, Vinicius Carvalho Lopes

AI总结:

本研究实证评估了Codex、GPT-3.5-Turbo和StarCoder三个模型在Java单元测试生成中的表现,发现其在HumanEval上覆盖率较高但在EvoSuite SF110上极低,且生成的测试存在测试坏味。

摘要:

代码生成模型通过接收来自代码注释、现有代码或两者结合的提示来生成代码。尽管代码生成模型(例如GitHub Copilot)在实践中日益被采用,但尚不清楚在不针对Java等强类型语言进行微调的情况下,它们能否成功用于单元测试生成。为填补这一空白,我们研究了三个模型(Codex、GPT-3.5-Turbo和StarCoder)生成单元测试的能力。我们使用两个基准(HumanEval和Evosuite SF110)来研究上下文生成对单元测试生成过程的影响。我们基于编译率、测试正确性、测试覆盖率和测试坏味来评估模型。我们发现Codex模型在HumanEval数据集上实现了超过80%的覆盖率,但没有模型在EvoSuite SF110基准上的覆盖率超过2%。生成的测试也存在测试坏味,如重复断言和空测试。

英文摘要:

A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑