超越孤立任务：一个评估编码代理在连续软件演进中的框架
Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution
AI总结：
本文提出一个框架，通过模拟真实开发者工作流程的两个现实场景，评估编码代理在连续任务中的表现，揭示孤立任务评估的局限性及多维评估的必要性。
摘要：
现有用于评估编码代理的数据集仅在无状态方式下评估孤立的单个拉取请求（PR）任务，无法捕捉现实世界软件开发中代码变更累积、技术债务积累和测试套件增长的现实。为弥合这一差距，我们引入了一个自动编码任务生成框架，帮助生成我们的数据集SWE-STEPS，通过两个现实场景评估编码代理在长周期任务中的表现：具有迭代请求的对话式编码，以及基于单次项目需求文档（PRD）的编码。与现有仅评估孤立拉取请求（PR）的数据集不同，我们的框架评估了依赖PR链的性能，使能够评估连续执行、回归验证和长期仓库健康。我们发现，广泛使用的孤立PR评估在我们的设置下会产生虚高成功率，可能高估性能多达20个百分点，因为它们忽略了先前低效或错误代码的“溢出”效应。此外，我们的分析表明，即使代理成功解决了一些问题，它们也会通过生成更高认知复杂度和技术债务的代码来降低仓库健康，凸显了多维评估的必要性。
英文摘要：
Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
