We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.
Pull Requests as a Training Signal for Repo-Level Code Editing
King's College London, UK(伦敦国王学院);Chinese University of Hong Kong, HK(香港中文大学);National University of Singapore, SG(新加坡国立大学);Microsoft Research Asia, CN(微软亚洲研究院);The Alan Turing Institute, UK(艾伦·图灵研究所)
SWE Context Bench: A Benchmark for Context Learning in Coding
Independent Researcher(独立研究员);University of Oxford(牛津大学);University of Edinburgh(爱丁堡大学);Technical University of Munich(慕尼黑工业大学);MemoraX AI;National University of Singapore(新加坡国立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学)
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Abundant;Zenity;Harvard University(哈佛大学);University of Waterloo(滑铁卢大学);Gujarat Technological University(古吉拉特技术大学);Warping;Stanford University(斯坦福大学);UNC-Chapel Hill(北卡罗来纳大学教堂山分校);Independent(独立);Refresh;Soleda AI;Near AI;Georgia Tech(佐治亚理工学院);Comenius University in Bratislava(布拉迪斯拉发Comenius大学);UC San Diego(圣地亚哥大学);BenchFlow;UC Santa Barbara(圣巴巴拉大学)
Chronos Enables Code Agents to Reason over Software Evolution
Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Hithink Research(航信研究院);National Industrial Information Security Development Research Center(国家工业信息安全发展研究中心)