DiffTestGen：基于大语言模型的变更导向测试以揭示行为差异
DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences
AI总结：
研究如何确保软件行为变化符合预期，提出基于大语言模型的DiffTestGen方法，利用静态调用图分析等确定入口点并改进联合覆盖指标，实验表明该方法能有效揭示行为差异、提高覆盖率并检测回归错误。
AI中文摘要：
随着软件随时间演变，确保行为变化符合开发者预期很重要。一种有前景的方法是生成能揭示程序新旧版本行为差异的测试。然而，当前方法对许多代码变更无法触发行为差异。本文提出DiffTestGen，一种新颖的基于大语言模型的变更导向差异测试方法。它利用静态调用图分析和项目文档确定测试生成的有效入口点并引导大语言模型触及变更代码；还迭代改进新引入的联合覆盖指标。在两个数据集上评估，DiffTestGen能揭示78.2%的拉取请求中的行为差异，平均联合覆盖率达90.7%。与基线相比，它能多揭示99个拉取请求中的差异，代码覆盖率分别提高12.5%和15.6个百分点，还能用于检测现有最佳方法遗漏的回归错误。
英文摘要：
As software evolves over time, it is important to ensure that any behavioral changes occur as intended by developers. A promising approach for this goal is to generate tests that expose behavioral differences between the old and new versions of a program. However, current approaches fail to trigger behavioral differences for many code changes. This paper presents~DiffTestGen, a novel change-directed, LLM-based differential testing approach specifically designed to expose behavioral differences introduced by a code change. The approach is enabled by two key contributions: First, DiffTestGen leverages static call graph analysis and project documentation to identify valid entry points for test generation and to guide the LLM toward reaching the changed code. Second, DiffTestGen iteratively improves our newly introduced union coverage metric, which combines coverage of modified code in the old and the new version, by providing targeted coverage feedback to the LLM. We evaluate DiffTestGen on two datasets comprising a total of 463 PRs. DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%. Compared with the baselines, DiffTestGen exposes 99 more PRs overall and increases code coverage by 12.5% and 15.6% percentage points, respectively. By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.
