arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2311.04532cs.SE
收藏

评估多样化大语言模型用于自动化通用缺陷复现

Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction

Sungmin Kang, Juyeon Yoon, Nargiz Askarbekkyzy, Shin Yoo

备注：
This work is an extension of our prior work, available at arXiv:2209.11515
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对缺陷复现难以自动化的问题，提出结合LLM生成测试用例与后处理筛选的LIBRO技术，在Defects4J基准复现约三分之一缺陷，并评估多类LLM的复现性能差异与规模影响。

摘要：

缺陷复现是开发者的一项核心工作，同时也很难实现自动化，因为缺陷报告通常以自然语言撰写，难以稳定地转化为测试用例。因此，现有技术大多聚焦于崩溃类缺陷，这类缺陷更易于自动检测与验证。\n在本研究中，我们利用大语言模型（LLM）突破了这一限制，已有研究证明LLM擅长自然语言处理与代码生成。通过提示LLM生成可复现缺陷的测试用例，并借助后处理流水线自动筛选出有潜力的生成测试，我们提出的技术LIBRO能够在广泛使用的Defects4J基准上成功复现约三分之一的缺陷。\n此外，我们对15款LLM（包括11款开源LLM）开展了全面评估，结果显示开源LLM同样具备巨大潜力：在大型Defects4J基准上，StarCoder LLM的复现性能达到闭源OpenAI LLM code-davinci-002的70%；在一个大概率未被任何LLM纳入训练数据的留出缺陷数据集上，其性能达到后者的90%。另外，我们针对不同规模LLM的实验表明，随着LLM规模增大，使用LIBRO的缺陷复现效果会提升，这为哪些LLM可适配LIBRO流水线提供了参考依据。

英文摘要：

Bug reproduction is a critical developer activity that is also challenging to automate, as bug reports are often in natural language and thus can be difficult to transform to test cases consistently. As a result, existing techniques mostly focused on crash bugs, which are easier to automatically detect and verify. In this work, we overcome this limitation by using large language models (LLMs), which have been demonstrated to be adept at natural language processing and code generation. By prompting LLMs to generate bug-reproducing tests, and via a post-processing pipeline to automatically identify promising generated tests, our proposed technique LIBRO could successfully reproduce about one-third of all bugs in the widely used Defects4J benchmark. Furthermore, our extensive evaluation on 15 LLMs, including 11 open-source LLMs, suggests that open-source LLMs also demonstrate substantial potential, with the StarCoder LLM achieving 70% of the reproduction performance of the closed-source OpenAI LLM code-davinci-002 on the large Defects4J benchmark, and 90% of performance on a held-out bug dataset likely not part of any LLM's training data. In addition, our experiments on LLMs of different sizes show that bug reproduction using LIBRO improves as LLM size increases, providing information as to which LLMs can be used with the LIBRO pipeline.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑