TestGenEval：一个真实世界单元测试生成与测试补全基准
TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
AI总结：
本研究发布了大规模测试生成基准TestGenEval，基于SWEBench构建，覆盖三类测试任务，评估显示主流模型在高覆盖率测试生成上表现不佳，为测试生成性能的全面评估提供了支撑。
摘要：
代码生成模型可助力改进从代码补全到缺陷预测等诸多常见软件任务。现有大多数面向代码生成大语言模型（LLM）的基准都聚焦于代码编写或代码补全。令人意外的是，尽管经过充分测试的软件与有效的缺陷检测之间存在强相关性，针对软件测试的基准构建工作却少得多。为填补这一空白，我们构建并发布了TestGenEval，这是一个用于衡量测试生成性能的大规模基准。TestGenEval基于SWEBench构建，包含来自11个维护良好的Python代码仓库中1210组代码与测试文件对的68647个测试用例，覆盖初始测试编写、测试套件补全以及代码覆盖率提升三类任务。测试编写任务模拟开发者从零编写测试套件的过程，测试补全任务则模拟开发者旨在提升现有测试套件覆盖率的场景。我们评估了多个参数规模从7B到405B不等的主流模型。详细分析表明，TestGenEval有助于实现对测试生成性能的全面评估。特别值得注意的是，模型在生成高覆盖率测试套件方面存在困难，表现最佳的模型GPT-4o的平均覆盖率仅为35.2%。这主要是因为模型难以对执行过程进行推理，且在处理复杂代码路径时频繁出现断言错误。
英文摘要：
Code generation models can help improve many common software tasks ranging from code completion to defect prediction. Most of the existing benchmarks for code generation LLMs focus on code authoring or code completion. Surprisingly, there has been far less effort dedicated to benchmarking software testing, despite the strong correlation between well-tested software and effective bug detection. To address this gap, we create and release TestGenEval, a large-scale benchmark to measure test generation performance. Based on SWEBench, TestGenEval comprises 68,647 tests from 1,210 code and test file pairs across 11 well-maintained Python repositories. It covers initial tests authoring, test suite completion, and code coverage improvements. Test authoring simulates the process of a developer writing a test suite from scratch, while test completion mimics the scenario where a developer aims to improve the coverage of an existing test suite. We evaluate several popular models, with sizes ranging from 7B to 405B parameters. Our detailed analysis highlights TestGenEval's contribution to a comprehensive evaluation of test generation performance. In particular, models struggle to generate high-coverage test suites, with the best model, GPT-4o, achieving an average coverage of only 35.2%. This is primarily due to models struggling to reason about execution, and their frequent assertion errors when addressing complex code paths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
