超越分辨率率:编码代理成功与失败的行为驱动因素
Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
AI总结:
本文通过大规模实证研究分析了9374条轨迹,探讨了编码代理在特定任务中失败的原因,发现行为模式和LLM能力对成功与失败有显著影响。
摘要:
编码代理代表了自动化软件工程的新范式,结合了大型语言模型(LLMs)的推理能力与工具增强的交互循环。然而,编码代理仍存在严重限制。排名靠前的基于LLM的编码代理在超过20%的基准问题上仍失败。我们缺乏对失败原因(即原因)以及失败如何行为化系统理解。我们进行了大规模实证研究,分析了19个代理(8个编码代理框架,14个LLMs)在500个任务上的9374条轨迹。我们的分析围绕三个研究问题展开。首先,我们调查了代理在特定任务中失败的原因,发现补丁复杂性本身并不能解释难度:12个从未解决的任务只需要简单的补丁,并被人类注释者认为是容易的,但所有代理都因架构推理和领域知识的缺口而失败。其次,我们考察了行为模式如何区分成功与失败。广泛报道的轨迹长度与失败的相关性在任务难度被控制后方向反转,揭示其为干扰因素。相反,轨迹结构一致地区分了成功与失败:在编辑前收集上下文并投资于验证的代理更常成功,这些策略由代理决定而非任务适应。第三,我们解开了LLM能力与框架设计,发现LLM是成功与行为的主驱动因素:共享相同LLM的代理在更多任务上达成一致,而框架性能差距随着LLM每一代的进步而缩小。框架提示确实影响代理战术,但这种影响随着更强的LLM而减弱。
英文摘要:
Coding agents represent a new paradigm in automated software engineering, combining the reasoning capabilities of Large Language Models (LLMs) with tool-augmented interaction loops. However, coding agents still have severe limitations. Top-ranked LLM-based coding agents still fail on over 20% of benchmarked problems. Yet, we lack a systematic understanding of why (i.e., the causes) agents fail, and how failure unfolds behaviorally. We present a large-scale empirical study analyzing 9,374 trajectories from 19 agents (8 coding agent frameworks, 14 LLMs) on 500 tasks. We organize our analysis around three research questions. First, we investigate why agents fail on specific tasks and find that patch complexity alone does not explain difficulty: 12 never-solved tasks require only simple patches and were considered easy by human annotators, yet all agents fail due to gaps in architectural reasoning and domain knowledge. Second, we examine how behavioral patterns differentiate success from failure. The widely reported correlation between trajectory length and failure reverses direction once task difficulty is controlled, revealing it as a confound. Instead, trajectory structure discriminates consistently: agents that gather context before editing and invest in validation succeed more often, and these strategies are agent-determined rather than task-adaptive. Third, we disentangle LLM capability from framework design and find that the LLM is the primary driver of both outcome and behavior: agents sharing the same LLM agree on far more tasks than agents sharing the same framework, and the framework performance gap shrinks with each generation of LLM improvement. Framework prompts do influence agent tactics, but this influence diminishes with stronger LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。