arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2408.02232cs.SEcs.AI
收藏

SpecRover:通过大语言模型进行代码意图提取

SpecRover: Code Intent Extraction via LLMs

National University of Singapore(新加坡国立大学)

Haifeng Ruan, Yuntong Zhang, Abhik Roychoudhury

AI总结:

本研究提出基于AutoCodeRover的SpecRover方法,通过迭代代码搜索与规约推断提取代码意图,经评审智能体审核补丁,在SWE-Bench上效能提升超50%且成本较低,验证了规约推断在LLM时代自动程序修复的重要性。

摘要:

自主程序改进通常涉及自动生成错误修复和功能新增。这类程序改进可通过大语言模型(LLM)与程序分析能力相结合,以LLM智能体的形式实现。由于程序修复或程序改进通常需要预期行为的规约,因此规约推断对于生成高质量的程序补丁十分有用。\n在本研究中,我们探究了LLM智能体内部用于迭代规约推断的高效低成本工作流。针对软件项目中待解决的GitHub issue,我们的目标是开展伴随规约推断的迭代代码搜索,从而从项目结构和行为两方面推断意图。\n由此捕获的意图会由评审智能体进行检查,目的是审核补丁并提供对已审核补丁的置信度度量。我们提出的方法SpecRover(AutoCodeRover-v2)基于开源LLM智能体AutoCodeRover构建。在包含2294个GitHub issue的完整SWE-Bench上的评估显示,其效能较AutoCodeRover提升了50%以上。与现有开源智能体相比,我们的工作在解决SWE-Bench lite中平均每个GitHub issue的解决成本较低(每个issue 0.65美元)。\nSpecRover生成的解释能够为开发者提供更好的“信号”,告知何时可以放心接受建议的补丁。SpecRover还旨在证明,即便程序修复技术进入LLM时代,规约推断在自动程序修复中依然具有持续的重要性。

英文摘要:

Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑