统一并制胜:利用 LLM 进行多语言、多样化且自洽的单元测试生成
Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs
AI总结:
本文提出 PolyTest,通过跨语言生成与高温多样采样并统一测试集,利用 LLM 的自洽性消除矛盾测试,在 EvalPlus 多模型实验中显著提升覆盖率、变异得分和通过率。
摘要:
基于大型语言模型(LLM)的测试生成已在软件工程领域受到关注,但大多数研究评估的是 LLM 针对给定语言在单次尝试中生成单元测试的能力,错失了利用 LLM 多样性实现更稳健测试的机会。本文提出 PolyTest,这是一种通过利用多语言和温度控制的多样性来增强测试生成的新方法。PolyTest 以两种互补方式系统性地利用这些特性:(1)跨语言测试生成,即在零温度下以多种语言生成测试,随后进行统一;(2)多样化测试采样,即在统一之前以较高温度在同一语言内生成多个测试集。一个关键洞见是,LLM 可能在不同语言和不同生成结果之间生成多样但相互矛盾的测试——相同输入、不同预期输出。PolyTest 通过统一测试集来缓解不一致性,促进自洽性并提升整体测试质量。与单语言或单次尝试方法不同,PolyTest 无需即时执行即可增强测试,这使其对表现较弱的语言尤其有益。我们使用 EvalPlus 在 Llama3-70B、GPT-4o 和 GPT-3.5 上评估 PolyTest,在温度 0 下以五种语言(Java、C、Python、JavaScript 和一种基于 CSV 的格式)生成测试,并在温度 1 下采样多个测试集。我们观察到,LLM 在各种设置下频繁生成相互矛盾的测试,而 PolyTest 在所有考虑的指标上都显著提升了测试质量——测试数量、通过率、语句/分支覆盖率(最高提升 9.01%)以及变异得分(最高提升 11.23%)。最后,PolyTest 在测试生成、通过率和变异得分方面优于 Pynguin。
英文摘要:
Large language model (LLM)-based test generation has gained attention in software engineering, yet most studies evaluate LLMs' ability to generate unit tests in a single attempt for a given language, missing the opportunity to leverage LLM diversity for more robust testing. This paper introduces PolyTest, a novel approach that enhances test generation by exploiting polyglot and temperature-controlled diversity. PolyTest systematically leverages these properties in two complementary ways: (1) Cross-lingual test generation, where tests are generated in multiple languages at zero temperature and then unified; (2) Diverse test sampling, where multiple test sets are generated within the same language at a higher temperature before unification. A key insight is that LLMs can generate diverse yet contradicting tests -- same input, different expected outputs -- across languages and generations. PolyTest mitigates inconsistencies by unifying test sets, fostering self-consistency and improving overall test quality. Unlike single-language or single-attempt approaches, PolyTest enhances testing without requiring on-the-fly execution, making it particularly beneficial for weaker-performing languages. We evaluate PolyTest on Llama3-70B, GPT-4o, and GPT-3.5 using EvalPlus, generating tests in five languages (Java, C, Python, JavaScript, and a CSV-based format) at temperature 0 and sampling multiple sets at temperature 1. We observe that LLMs frequently generate contradicting tests across settings, and that PolyTest significantly improves test quality across all considered metrics -- number of tests, passing rate, statement/branch coverage (up to +9.01%), and mutation score (up to +11.23%). Finally, PolyTest outperforms Pynguin in test generation, passing rate, and mutation score.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。