arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.08588cs.SE
收藏

LLM vs. 人类单元测试:对真实 Python 错误的故障检测

LLM vs. Human Unit Tests: Fault Detection on Real Python Bugs

University of Tennessee(田纳西大学)

Phouvadeth Vathana, Prapti Bhatt, Rishi Patel, Nasir U. Eisty

AI总结:

通过对比 LLM 生成与人工编写的单元测试在真实 Python 错误上的故障检测能力,发现 LLM 结合检索增强上下文可检测 69% 的故障,远超人工测试的 17.2%,且代码覆盖率相近,表明覆盖率不足以衡量故障检测能力。

摘要:

大型语言模型(LLM)在自动化单元测试生成方面显示出巨大潜力,但其相对于人工编写测试的实际有效性仍不明确。现有评估通常依赖面向覆盖率的基准测试,无法直接评估故障检测能力。我们在三个互补的 Python 基准测试上对 LLM 生成和人工编写的单元测试进行了实证比较:来自 BugsInPy 的 29 个真实历史错误、一个基于 python-slugify 和 packaging 的函数级基准测试,以及一个受控配对基准测试。我们的生成流程将 Gemini 2.5 Flash 与轻量级词汇检索机制相结合,在生成时提供与错误相关的上下文。在八个质量维度上,使用检索增强上下文的 LLM 生成测试在 69% 的案例中检测到故障,而通用人工编写测试仅为 17.2%(Fisher 精确检验,$p < 0.001$,Cohen's $h = 1.10$)。关键的是,两种方法之间的行覆盖率和分支覆盖率几乎相同(84.8% vs. 88.5% 和 75.2% vs. 82.1%),证实覆盖率不足以作为故障检测能力的代理指标。我们讨论了每种方法擅长的条件,描述了它们的互补优势,并指出了检索上下文和可重复基准测试构建在有意义的测试质量评估中的关键作用。

英文摘要:

Large language models (LLMs) have shown considerable promise for automated unit test generation, yet their practical effectiveness relative to human-written tests remains poorly understood. Existing evaluations commonly rely on coverage-oriented benchmarks that do not assess fault-detection capability directly. We present an empirical comparison of LLM-generated and human-written unit tests across three complementary Python benchmarks: 29 real historical bugs from BugsInPy, a function-level benchmark drawn from python-slugify and packaging, and a controlled paired benchmark. Our generation pipeline couples Gemini 2.5 Flash with a lightweight lexical retrieval mechanism that supplies bug-relevant context at generation time. Across eight quality dimensions, LLM-generated tests with retrieval-augmented context detect faults in 69% of cases compared to 17.2% for general-purpose human-written tests (Fisher's exact, $p < 0.001$, Cohen's $h = 1.10$). Critically, line and branch coverage are nearly identical between the two approaches (84.8% vs. 88.5% and 75.2% vs. 82.1%), confirming that coverage is an insufficient proxy for fault-detection capability. We discuss the conditions under which each approach excels, characterize their complementary strengths, and identify the critical role of retrieval context and reproducible benchmark construction in meaningful test-quality evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑