SWE-Bench Pro Verified：面向软件工程智能体的可靠基准

中文摘要 AI 辅助

SWE-Bench Pro 已成为在具有挑战性的仓库级任务上评估软件工程智能体的标准基准。然而，我们的分析工作表明，其评估受到两个不可靠来源的削弱：\textbf{奖励黑客}，由金标准解决方案或隐藏评估信息的泄露所致，以及\textbf{任务质量问题}，包括误导性的问题陈述和范围不当的测试。这些问题可能夸大基准性能，并掩盖智能体真实的编码能力。我们提出 \textbf{SWE-Bench Pro Verified}，这是 SWE-Bench Pro 的一个经过验证的版本，解决了这两个问题。我们的方法结合了\textbf{反黑客}保障措施，在不干扰正常智能体功能的情况下消除主要泄露渠道，以及\textbf{任务精炼}，对缺陷实例中的不一致之处进行最小化修正。在 SWE-Bench Pro Verified 上的评估显示，一些模型的性能明显低于先前报告的结果，表明 SWE-Bench Pro 上现有的结果可能高估了真实的软件工程能力。SWE-Bench Pro Verified 为评估软件工程智能体提供了一个更值得信赖的基准。

英文摘要

SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.

发表机构

• East China Normal University（华东师范大学）
• Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
• Fudan University（复旦大学）

机构由 AI 辅助整理，请以论文原文为准。