代码理解是编码智能体的瓶颈
Code Understanding is a Bottleneck for Coding Agents
浏览论文内容
中文总结 AI 辅助
该研究针对编码智能体评估的现有基准缺陷，提出CABRA合成评估框架，通过多维度任务规模参数构建6840个任务，发现代码理解是编码智能体的瓶颈，工具调用数比编辑行数更能预测智能体准确率。
中文摘要 AI 辅助
用于编码智能体的仓库基准（如SWE-bench）通常假设编辑的代码行数可预测任务难度，但这类数据集对代码和任务类型的控制不足，难以明确哪些能力真正导致智能体出错。我们提出CABRA：用于严格智能体评估的编码能力蓝图。CABRA从头构建任务，作为调用图转换，并通过四个维度的任务规模参数扩展难度：函数遍历、搜索、运行时解析和指令遵循。我们在6840个CABRA任务上运行8个大语言模型（LLM）和6个编码智能体，结果显示：1）LLM准确率随任务规模增大而下降，但智能体通过将工作卸载到工具（如grep）保持接近完美；2）更大的CABRA任务会引发更多用于读取和分析的工具调用，而对SWE-bench Verified的单独研究显示，这些工具调用数量比编辑的代码行数更能预测智能体的准确率，表明智能体的任务难度可能在于理解待编辑的代码，而非仅在于编辑操作；3）将CABRA扩展到让模型分析两类逻辑差异的高强度理解任务，支持了这一发现，此时智能体准确率终于下降。更广泛地说，我们主张采用CABRA这类合成评估，以揭示被工具掩盖的LLM弱点（如干草堆中的针）以及编码智能体仍觉困难的、超出编辑之外的能力（如理解），将SWE-bench式任务与受控诊断相结合。
英文摘要
Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present CABRA: a Coding Ability Blueprint for Rigorous Agent evaluation. CABRA builds tasks from scratch as call graph transformations and scales difficulty via a task size parameter on four axes: function traversal, search, runtime resolution, and instruction following. We run eight LLMs and six coding agents on 6,840 CABRA tasks to show: 1) LLM accuracy falls as task size~grows, but agents stay near-perfect by offloading work to tools (e.g., grep); 2) Larger CABRA tasks elicit more tool calls for reading and analysis, while a separate study on SWE-bench Verified shows these tool call counts predict agents' accuracy better than lines of code edited, suggesting task difficulty for agents can lie in understanding code to edit, not just in making edits; 3) Extending CABRA to an intense understanding task where models analyze divergent logic across two classes backs this finding, as agent accuracy finally falls. More broadly, we argue for synthetic evaluations like CABRA to unmask LLM weaknesses trivialized by tools (e.g., needle-in-a-haystack) and abilities beyond just editing (e.g., understanding) that coding agents still find difficult, pairing SWE-bench-style tasks with controlled diagnosis.
发表机构
- Microsoft Research（微软研究院）
- University of Maryland（马里兰大学）
机构由 AI 辅助整理，请以论文原文为准。
