错误代码对测试生成影响的度量
Measuring the Influence of Incorrect Code on Test Generation
AI总结：
本研究测量了正确与错误被测代码对LLM测试生成的影响，发现正确代码可提升准确性、覆盖率和缺陷检测，且自然语言描述能进一步改善效果。
摘要：
人们自然认为，与使用错误的被测代码相比，当大型语言模型（LLM）被提示使用正确的被测代码时，更有可能生成正确的测试用例。然而，尽管这一效应的大小对于软件工程师和研究人员都具有明显的重要性，但此前从未被测量过。为了回答这个问题，我们对5个开源和6个闭源语言模型进行了全面的实证研究，使用了3个广泛使用的基准数据集以及来自两个不同真实世界数据集的41个仓库级真实世界示例。我们的结果显示，与错误的被测代码相比，使用正确代码提示的LLM在测试准确性、代码覆盖率和缺陷检测方面分别实现了57%、12%和24%的提升。我们进一步表明，这些科学结论从三个基准数据集延续到真实世界代码，其中为错误代码生成的测试在缺陷检测率上表现差47%。最后，我们报告说，通过提供自然语言代码描述，可以实现准确性+18%、覆盖率+4%和缺陷检测+34%的提升。这些发现具有可操作的结论。例如，真实世界缺陷检测中47%的降低是一个明显的担忧。幸运的是，这是一个我们的关于描述附加价值的发现可以提供立即可行的补救措施的担忧。
英文摘要：
It is natural to suppose that a Large Language Model is more likely to generate correct test cases when prompted with correct code under test, compared to incorrect code under test. However, the size of this effect has never been previously measured, despite its obvious importance for both practicing software engineers and researchers. To answer the question, we conducted a comprehensive empirical study on 5 open source and 6 closed source language models, with 3 widely-used benchmark data sets together with 41 repo-level real-world examples from two different real-world data sets. Our results reveal that, when compared to incorrect code under test, LLMs prompted with correct code achieve improvements in test accuracy, code coverage, and bug detection of 57\%, 12\%, and 24\% respectively. We further show that these scientific conclusions carry over from the three benchmark data sets to the real-world code, where tests generated for incorrect code experience a 47\% worse bug detection rate. Finally, we report that improvements of +18\% in accuracy, +4\% coverage, and +34\% in bug detection can be achieved by providing natural language code descriptions. These findings have actionable conclusions. For example, the 47\% reduction in real-world bug detection is a clear concern. Fortunately, it is a concern for which our findings about the added value of descriptions offer an immediately actionable remedy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
