arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.00385cs.SEcs.LG
收藏

让对话继续:使用 ChatGPT 以每个 0.42 美元的成本修复 337 个 bug 中的 162 个

Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Chunqiu Steven Xia, Lingming Zhang

AI总结:

本文提出对话驱动的自动程序修复方法 ChatRepair,利用测试失败反馈和历史补丁成功经验迭代生成补丁,并基于 ChatGPT 以低成本修复 337 个 bug 中的 162 个。

摘要:

自动程序修复(Automated Program Repair,APR)旨在为有缺陷的程序自动生成补丁。近期的 APR 工作集中于利用现代大型语言模型(Large Language Models,LLM)直接为 APR 生成补丁。这类基于 LLM 的 APR 工具的工作方式是,首先使用原始有缺陷代码构建输入提示,然后查询 LLM 以生成补丁。尽管基于 LLM 的 APR 工具能够取得最先进的结果,但它仍然遵循经典的“生成并验证”修复范式,即先生成大量补丁,随后再逐一验证每个补丁。这不仅会产生许多重复且不正确的补丁,还会遗漏测试失败以及看似合理的补丁中包含的关键信息。为解决这些限制,我们提出了 ChatRepair,这是首个完全自动化、由对话驱动的 APR 方法,它将补丁生成与即时反馈交替进行,以对话方式执行 APR。ChatRepair 首先向 LLM 提供相关的测试失败信息作为起点,然后从同一 bug 早期补丁尝试的失败与成功中学习,以实现更强大的 APR。对于未能通过所有测试的早期补丁,我们将不正确的补丁与其对应的相关测试失败信息结合起来,为 LLM 构建新的提示以生成下一个补丁。通过这种方式,我们可以避免犯同样的错误。对于通过了所有测试的早期补丁,我们进一步要求 LLM 生成原始看似合理补丁的替代变体。通过这种方式,我们可以进一步基于早期成功并从中学习,生成更多看似合理的补丁,从而提高获得正确补丁的机会。尽管我们的方法具有通用性,但我们使用最先进的基于对话的 LLM——ChatGPT 来实现 ChatRepair。通过计算访问 ChatGPT 的成本,我们能够以每个 0.42 美元的代价修复 337 个 bug 中的 162 个!

英文摘要:

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then queries the LLM to generate patches. While the LLM-based APR tools are able to achieve state-of-the-art results, it still follows the classic Generate and Validate repair paradigm of first generating lots of patches and then validating each one afterwards. This not only leads to many repeated patches that are incorrect but also miss the crucial information in test failures as well as in plausible patches. To address these limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests, we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM -- ChatGPT. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for \$0.42 each!

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑