立场：编程基准与智能体软件工程不一致
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
AI总结：
本文指出当前编程基准在智能体时代存在三大问题：混淆模型与系统框架、单一参考答案惩罚有效替代方案、缺乏组件级信号导致迭代困难，并提出应重新设计基准以对齐智能体软件工程。
摘要：
编程智能体已成为软件工程的主要模式，但我们用于比较它们的基准是在智能体时代之前设计的：它们将模型、框架和环境合并为一个单一的端到端分数，通常针对一个参考答案进行计算，没有提供用于迭代的组件级信号。我们认为当前的编程基准与智能体软件工程不一致。在实践中，编程智能体不是一个模型：它是一个系统框架——由模型、框架、上下文、环境和反馈信号组成的复合体，其中任何一个都可能使基准分数移动与相邻模型代际之间相当的幅度。我们讨论了三个症状：(i) 基准分数混淆了模型与框架的其余部分；(ii) 针对单一参考答案评分惩罚了同样有效的替代方案；(iii) 缺乏单个框架组件级别的信号使得端到端系统分数难以迭代。
英文摘要：
Coding agents have become a major mode of software engineering, but the benchmarks we use to compare them were designed in a pre-agent era: they collapse model, harness, and environment into a single end-to-end score, typically computed against one reference solution, with no component-level signal for iteration. We argue that current coding benchmarks are misaligned with agentic software engineering. A coding agent in practice is not a model: it is a system harness -- a composite of models, harnesses, contexts, environments, and feedback signals, any one of which can move the benchmark score by margins comparable to those between adjacent model generations. We discuss three symptoms: (i) benchmark scores conflate the model with the rest of the harness; (ii) grading against a single reference solution penalises equally valid alternatives; and (iii) the absence of signal at the level of individual harness components makes the end-to-end system score difficult to iterate on.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
