AI 中文总结
研究CLI编码代理失败轨迹，引入面向过程框架，收集并标注大量执行轨迹，发现失败多由认知错误驱动，起始于最初几步且常隐藏，提出提高编码代理可靠性需早期验证和干预，而非仅靠最终结果评估。
AI 中文摘要
大语言模型编码代理越来越多地被部署在基于终端的环境中自主执行软件工程任务，其可靠性备受关注。现有实证研究多将失败视为最终结果而非时间过程，对失败如何出现、演变及不可恢复了解有限。本文进行了首个关于CLI编码代理失败轨迹的大规模实证研究，引入面向过程的框架，通过执行轨迹的起始、演变和恢复来分析失败。先收集了3843条执行轨迹，经筛选得到1794条完整有效轨迹进行人工标注（超63000个执行步骤），得出14项发现。结果表明编码代理失败主要由认知错误驱动，多在最初几步开始，常隐藏直至无法恢复，这意味着提高可靠性需早期验证和干预，而非仅依赖最终结果评估。
英文摘要
Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.
Comments12 pages, 6 figures
