arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.02883cs.SEcs.CLcs.LG
收藏

TDD-Bench Verified:LLM 能否在问题解决前为其生成测试?

TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?

IBM Research(IBM研究院)

Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, Saurabh Sinha

AI总结:

本文针对TDD缺乏自动化测试生成的问题,构建含449个真实问题的TDD-Bench Verified基准套件,提出Auto-TDD解决方案,其“先失败后通过”率及覆盖充分性优于现有工作,助力开发高效解决问题并提升修复稳健性。

摘要:

测试驱动开发(TDD)是先写测试再编码的实践,TDD 的支持者阐述了其诸多益处。例如,针对源代码仓库中的某个问题,测试可在相关人员为商定的修复方案编写代码前,在利益相关者之间明确期望的行为。尽管针对“先写代码后测试”的实践已有大量自动化测试生成工作,但针对 TDD 的此类自动化却很少。理想情况下,TDD 的测试应是“先失败后通过”(即问题解决前失败,解决后通过),且对问题解决过程中变更的代码具有良好的覆盖充分性。本文介绍 TDD-Bench Verified,这是一个从真实世界 GitHub 代码仓库中挖掘的 449 个问题的高质量基准套件。该基准的评估工具仅孤立运行相关测试,以实现简单而准确的覆盖度量,且其数据集经过人工判断和工具执行的双重过滤。本文还提出 Auto-TDD,这是一种基于 LLM 的解决方案,输入问题描述和代码库(问题解决前的版本),输出可用于验证问题解决所做变更的测试。我们的评估表明,Auto-TDD 比最先进的现有工作具有更高的“先失败后通过”率,同时也具有更高的覆盖充分性。总体而言,我们希望这项工作能帮助开发人员更高效地解决问题,同时促成更稳健的修复。

英文摘要:

Test-driven development (TDD) is the practice of writing tests first and coding later, and the proponents of TDD expound its numerous benefits. For instance, given an issue on a source code repository, tests can clarify the desired behavior among stake-holders before anyone writes code for the agreed-upon fix. Although there has been a lot of work on automated test generation for the practice "write code first, test later", there has been little such automation for TDD. Ideally, tests for TDD should be fail-to-pass (i.e., fail before the issue is resolved and pass after) and have good adequacy with respect to covering the code changed during issue resolution. This paper introduces TDD-Bench Verified, a high-quality benchmark suite of 449 issues mined from real-world GitHub code repositories. The benchmark's evaluation harness runs only relevant tests in isolation for simple yet accurate coverage measurements, and the benchmark's dataset is filtered both by human judges and by execution in the harness. This paper also presents Auto-TDD, an LLM-based solution that takes as input an issue description and a codebase (prior to issue resolution) and returns as output a test that can be used to validate the changes made for resolving the issue. Our evaluation shows that Auto-TDD yields a better fail-to-pass rate than the strongest prior work while also yielding high coverage adequacy. Overall, we hope that this work helps make developers more productive at resolving issues while simultaneously leading to more robust fixes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑