代码仓库中的针:一项评估AI生成仓库编辑可维护性的基准测试
Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits
AI总结:
本文提出NITR基准测试,评估AI生成代码是否保持可维护结构,发现现有系统在复杂架构修改上表现不佳,代理模式提升性能但未解决根本问题。
摘要:
AI编码代理如今能完成复杂编程任务,但现有评估主要关注行为正确性而忽视可维护性风险。本文提出Needle in the Repo (NITR),一种诊断探针与 oracle 框架,用于评估行为正确的仓库编辑是否保持可维护结构。NITR将软件工程智慧提炼为嵌入小型真实多文件代码库的受控探针,每个代码库设计成成功依赖单一可维护性维度。每个探针配对隐藏评估工具,结合功能测试和结构 oracle,编码目标可维护性约束并返回可解释诊断。使用NITR,我们评估了23种编码配置在GPT、Claude、Gemini和Qwen家族中的表现。当前AI编码系统仍远未稳健:平均仅解决36.2%的案例,最佳达57.1%,且在多步骤案例中性能下降至20.6%。最严峻的挑战是架构而非局部编辑,尤其是依赖控制(4.3%)和责任分解(15.2%)。此外,64/483结果(13.3%)通过所有功能测试但失败结构 oracle。在我们的工具下,代理模式配置将平均性能从28.2%提升至45.0%,但未消除这些架构失败。这些结果表明,代码生成的进步尚未转化为可维护代码演进的进步,且NITR揭示了传统评估所遗漏的关键失败面。
英文摘要:
AI coding agents can now complete complex programming tasks, but existing evaluations largely emphasize behavioral correctness and often overlook maintainability risks such as weak modularity or testability. We present Needle in the Repo (NITR), a diagnostic probe-and-oracle framework for evaluating whether behaviorally correct repository edits preserve maintainable structure. NITR distills recurring software engineering wisdom into controlled probes embedded in small, realistic multi-file codebases, each designed so that success depends primarily on one targeted maintainability dimension. Each probe is paired with a hidden evaluation harness that combines functional tests for required behavior with structural oracles that encode the targeted maintainability constraint and return interpretable diagnoses. Using NITR, we evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings. Current AI coding systems remain far from robust: on average, configurations solve only 36.2% of cases, the best reaches 57.1%, and performance drops from 53.5% on micro cases to 20.6% on multi-step cases. The hardest pressures are architectural rather than local edits, especially dependency control (4.3%) and responsibility decomposition (15.2%). Moreover, 64/483 outcomes (13.3%) pass all functional tests yet fail the structural oracle. Under our harness, agent-mode configurations improve average performance from 28.2% to 45.0%, but do not eliminate these architectural failures. These results show that progress in code generation is not yet progress in maintainable code evolution, and that NITR exposes a critical failure surface missed by conventional evaluation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。