arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.26130cs.SEcs.AI
收藏

SWE-PRBench:基于拉取请求反馈的AI代码审查质量基准测试

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Deepak Kumar

AI总结:

SWE-PRBench通过350个带人工标注的拉取请求评估AI代码审查质量,发现即使在代码生成基准测试中表现良好,AI在仅基于差异的配置下仍只能检测出人类标记问题的15-31%,表明AI代码审查距离人类专家表现仍有差距。

摘要:

我们介绍了SWE-PRBench,一个包含350个拉取请求的基准测试集,配有人工标注的地面真实数据,用于评估AI代码审查质量。在经过验证的LLM-as-judge框架(kappa=0.75)评估下,8个前沿模型在仅差异配置下只能检测出人类标记问题的15-31%,表明尽管在代码生成基准测试中表现良好,AI代码审查仍远低于人类专家性能。拉取请求来自活跃的开源仓库,通过700个候选项目筛选,使用仓库质量评分过滤,并在三种冻结上下文配置下评估:仅差异(config_A)、差异加文件内容(config_B)和完整上下文(config_C),从而系统地消融上下文提供策略。所有8个模型从config_A到config_C单调退化,即使上下文通过结构化的语义层(包括提取的AST函数上下文和导入图解析)提供。主导机制是config_B中Type2_上下文问题检测的崩溃,与长上下文中的注意力稀释一致:一个结构化的2000个标记差异加摘要提示在所有8个模型中均优于一个2500个标记的完整上下文提示,后者增加了执行上下文、行为映射和测试签名。前四名模型在统计上无法区分(平均分数0.147-0.153),而与其他四个模型之间存在明显的层级差距(平均分数<=0.113)。数据集、上下文、标注和评估工具已公开发布。

英文摘要:

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑