发表机构
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出代码智能体过程评估的测量框架，用SCAE实现步骤级因果归因，经499个文件定位场景实验发现当前过程评估多测量语义相关性而非因果贡献。
AI 中文摘要
代码智能体的评估日益不仅关注其是否解决任务，还关注其执行方式。然而，现有的过程级评估常将动作预测、任务不确定性和步骤归因视为同一问题，导致这类评估实际测量的内容不明确。本文提出代码智能体过程评估的测量框架，并通过SCAE实例化步骤级因果归因，SCAE是基于智能体执行结构因果模型推导的、基于重放的估计器。该框架结合前缀条件识别、基于重放/干预的估计以及受控评判者信息操纵，以在动作、任务和步骤层级研究过程评估。对来自12个仓库的499个文件定位场景的实验表明，下一个动作主要由执行来源而非代码图转换驱动；执行不确定性在任务层级而非步骤层级具有结构性；完整轨迹评判者存在系统性碰撞偏差，这表明当前过程评估常测量语义相关性而非经认证的因果贡献。
英文摘要
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.
Comments38 pages, 8 figures
