arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.01616cs.SE
收藏

通过结合大语言模型和基于搜索的软件测试生成问题复现测试

Automated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing

University of Zurich(苏黎世大学);Mozilla Corporation(Mozilla公司)

Konstantinos Kitsios, Marco Castelluccio, Alberto Bacchelli

AI总结:

BLAST通过结合大语言模型和基于搜索的软件测试,自动从问题-补丁对中生成问题复现测试,提升测试覆盖率和补丁有效性

摘要:

问题复现测试在buggy代码上失败,在应用补丁后通过,从而增加开发人员的信心,认为问题已被解决且不会重新引入。然而,过去的研究表明,开发人员经常在没有此类测试的情况下提交补丁,这使得自动生成问题复现测试成为一个研究领域。我们提出BLAST,一种通过结合大语言模型和基于搜索的软件测试(SBST)从问题-补丁对中自动生成问题复现测试的工具。对于大语言模型部分,我们通过git历史分析、静态分析和SBST生成的测试提取相关上下文,补充问题描述和补丁。对于SBST部分,我们适应SBST生成问题复现测试;问题描述和补丁通过中间大语言模型生成的种子输入到SBST优化中,然后将其反序列化为SBST兼容的形式。BLAST成功地为一个经过整理的Python基准中的151/426(35.4%)的问题生成了问题复现测试,优于当前最先进的方法(23.5%)。此外,为了衡量BLAST的实际影响,我们构建了一个GitHub机器人,每当有新的pull request(PR)链接到问题时运行BLAST,如果BLAST生成了问题复现测试,该机器人会在PR中提出建议。我们将在三个开源仓库中部署该机器人三个月,收集了32个PR-问题对的数据。BLAST在这些情况下生成了11个问题复现测试,我们将其提交给开发人员。通过分析开发人员的反馈,我们讨论了研究人员和工具构建者的挑战和机会。数据和材料:https://doi.org/10.5281/zenodo.16949042

英文摘要:

Issue-reproducing tests fail on buggy code and pass once a patch is applied, thus increasing developers' confidence that the issue has been resolved and will not be re-introduced. However, past research has shown that developers often commit patches without such tests, making the automated generation of issue-reproducing tests an area of interest. We propose BLAST, a tool for automatically generating issue-reproducing tests from issue-patch pairs by combining LLMs and search-based software testing (SBST). For the LLM part, we complement the issue description and the patch by extracting relevant context through git history analysis, static analysis, and SBST-generated tests. For the SBST part, we adapt SBST for generating issue-reproducing tests; the issue description and the patch are fed into the SBST optimization through an intermediate LLM-generated seed, which we deserialize into SBST-compatible form. BLAST successfully generates issue-reproducing tests for 151/426 (35.4%) of the issues from a curated Python benchmark, outperforming the state-of-the-art (23.5%). Additionally, to measure the real-world impact of BLAST, we built a GitHub bot that runs BLAST whenever a new pull request (PR) linked to an issue is opened, and if BLAST generates an issue-reproducing test, the bot proposes it as a comment in the PR. We deployed the bot in three open-source repositories for three months, gathering data from 32 PRs-issue pairs. BLAST generated an issue-reproducing test in 11 of these cases, which we proposed to the developers. By analyzing the developers' feedback, we discuss challenges and opportunities for researchers and tool builders. Data and material: https://doi.org/10.5281/zenodo.16949042

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑