arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.23448cs.SEcs.AI
收藏

代码审查代理基准

Code Review Agent Benchmark

National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学)

Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, Abhik Roychoudhury

AI总结:

本文提出c-CRAB基准,用于评估代码审查代理的能力,发现现有代理仅能解决40%的任务,揭示了未来研究的潜力及人机协作的可能性。

摘要:

软件工程代理在编写代码方面展现出巨大潜力。随着AI代理渗透到代码编写中,并自动生成大量代码,代码质量成为关键问题。当自动生成的代码被整合到庞大的代码库中时,代码审查和广泛的质量保证变得重要。本文从新角度审视该问题,并精心整理了一个代码审查数据集,供AI代理使用。我们的数据集称为c-CRAB(发音为see-crab)可用于评估代理在代码审查任务中的能力。具体而言,给定一个拉取请求(可能来自代码生成代理或人类),如果代码审查代理生成审查,我们的评估框架可以评估代码审查代理的审查能力。我们的评估框架用于评估当前最先进的技术——开源PR代理,以及Devin、Claude Code和Codex等商业代码审查代理。我们的c-CRAB数据集系统地从人类审查中构建,给定一个拉取请求实例的人类审查,我们生成相应的测试来评估代码审查代理生成的审查。这种基准构建给我们提供了几个见解。首先,现有的审查代理共同只能解决约40%的c-CRAB任务,表明未来研究有潜力缩小这一差距。其次,我们发现代理审查往往考虑了与人类审查不同的方面,表明未来软件团队中人机协作的潜力。最后但同样重要的是,由我们的数据集生成的代理测试充当了预留的测试集,因此成为代理生成审查的质量门禁。未来代码生成代理、测试生成代理和代码审查代理之间的协作意义仍需进一步研究。

英文摘要:

Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑