SWE Atlas：超越问题解决的编码代理基准测试
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
AI总结：
SWE Atlas通过涵盖代码库问答、测试编写和重构三个软件工程流程，提供了一种新的基准测试套件，评估编码代理的正确性和工程质量。
摘要：
我们介绍了SWE Atlas，一个涵盖三个专业软件工程工作流的基准测试套件：代码库问答（124个任务）、测试编写（90个任务）和重构（70个任务）。SWE Atlas与以往的SWE基准测试不同，它针对那些被低估但实际重要的任务类别，使用了全面的类别特定评估协议，并采用了未明确规定、代理式任务形式，更贴近现实世界使用。其评估框架结合了程序性检查与基于标准的评估。这超出了功能正确性，评估软件工程质量，包括测试和重构的完整性、可维护性、可重用的抽象以及代码库的整洁性。我们评估了多种前沿和开放式模型在SWE Atlas上的表现，发现GPT-5.4和Opus 4.7在总体表现上最强，而即使最好的开放式模型得分也较差。我们的分析表明，顶级模型依赖于广泛的代码库探索和运行时驱动的推理。然而，即使顶级模型在细微边缘情况、复杂运行时分析和遵守软件工程最佳实践方面也持续遇到困难。总体而言，SWE Atlas为测量编码代理的正确性和工程质量提供了补充的评估套件。
英文摘要：
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
