arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.07531cs.SEcs.AI
收藏

在谷歌评估基于智能体的程序修复

Evaluating Agent-based Program Repair at Google

Google(谷歌)

Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, Satish Chandra

AI总结:

本文针对企业场景下的程序修复问题,构建了含178个谷歌缺陷的评估集,实现了适配谷歌环境的智能体Passerine,建立了工业级程序修复基准的性能基线。

摘要:

基于智能体的程序修复有望通过结合现代大语言模型(LLM)的规划、工具使用与代码生成能力,端到端自动解决复杂缺陷。近期研究已在流行的开源基准SWE-Bench(一个收录高星GitHub Python项目缺陷的集合)上探索了基于智能体的修复方法的应用,此外还提出了SWE-Agent等多种智能体方法来解决该基准中的缺陷问题。本文探讨了在企业场景下使用智能体方法修复缺陷的可行性。为此,我们从谷歌的问题追踪系统中整理了包含178个缺陷的评估集,该数据集涵盖78个人工报告的缺陷和100个机器报告的缺陷。\n 为了在该基准上建立修复性能基线,我们实现了Passerine——一个核心思路与SWE-Agent类似、可在谷歌开发环境中运行的智能体。实验表明,采用20条轨迹采样并搭配Gemini 1.5 Pro时,Passerine能为评估集中73%的机器报告缺陷和25.6%的人工报告缺陷生成通过缺陷测试的补丁(即看似合理的补丁)。经人工核查,我们发现43%的机器报告缺陷和17.9%的人工报告缺陷至少存在一个与真实补丁语义等价的补丁。\n 这些结果在一个与工业相关的基准上建立了基线,正如我们所展示的,该基准包含的缺陷在语言多样性、变更规模与分布等方面,与流行的SWE-Bench数据集中的缺陷来自不同的分布。

英文摘要:

Agent-based program repair offers to automatically resolve complex bugs end-to-end by combining the planning, tool use, and code generation abilities of modern LLMs. Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench, a collection of bugs from highly-rated GitHub Python projects. In addition, various agentic approaches such as SWE-Agent have been proposed to solve bugs in this benchmark. This paper explores the viability of using an agentic approach to address bugs in an enterprise context. To investigate this, we curate an evaluation set of 178 bugs drawn from Google's issue tracking system. This dataset spans both human-reported (78) and machine-reported bugs (100). To establish a repair performance baseline on this benchmark, we implement Passerine, an agent similar in spirit to SWE-Agent that can work within Google's development environment. We show that with 20 trajectory samples and Gemini 1.5 Pro, Passerine can produce a patch that passes bug tests (i.e., plausible) for 73% of machine-reported and 25.6% of human-reported bugs in our evaluation set. After manual examination, we found that 43% of machine-reported bugs and 17.9% of human-reported bugs have at least one patch that is semantically equivalent to the ground-truth patch. These results establish a baseline on an industrially relevant benchmark, which as we show, contains bugs drawn from a different distribution -- in terms of language diversity, size, and spread of changes, etc. -- compared to those in the popular SWE-Bench dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑