编码智能体未言明之事:衡量执行与报告间的任务状态差距
What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
AI总结:
本研究通过CTXGAP测量发现,编码智能体的报告与完整执行上下文间存在显著任务状态可恢复性差距,基于报告的恢复率远低于完整上下文,该差距在不同工具链及DeepSeek测试中均存在,CTXGAP因此确立了这一补充评估维度。
摘要:
人类主管依赖编码智能体的报告评估进度、验证修复情况并决定何时介入,但主管无法调查从未被告知发生的故障。失败尝试、变通方案和未解决的风险可能从常规报告中消失,导致主管忽略需审查的问题。我们开展CTXGAP(一项测量研究),量化智能体累积的面向用户报告与其完整记录的执行上下文之间的任务状态可恢复性差距。在37项编码任务和4799个有效探测中,智能体按需回答93.6%的任务状态问题;完整上下文读取者恢复92.7%,仅报告读取者恢复52.0%,差距达40.7个百分点。在30项已验证修复的最终检查点,差距仍为34.8个百分点。人工标注显示,提及却缺乏足够支持细节的状态是未恢复信息的最大类别,经模型评估的与监督相关的状态仍存在显著差距。完整终端滚动记录将恢复率提升至73.6%,但仍落后于完整上下文。在报告长度匹配的情况下,基于上下文的重写使恢复率提升约8个百分点,且未检测到针对性重写优于自然重写的优势。在不同工具链及独立DeepSeek的执行与测量中,均存在正向差距。CTXGAP将基于报告的任务状态可恢复性确立为一项补充评估维度。
英文摘要:
Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene. But a supervisor cannot investigate a failure they were never told occurred. Failed attempts, workarounds, and unresolved risks may disappear from routine reports, leaving supervisors unaware of issues that warrant scrutiny. We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts. Across 37 coding tasks and 4,799 valid probes, agents answer 93.6% of task-state questions on demand; full-context readers recover 92.7%, while report-only readers recover 52.0%, a 40.7-point gap. At the final checkpoints of 30 verified fixes, the gap remains 34.8 points. Human annotation identifies states mentioned without enough supporting detail as the largest category of unrecovered information, and model-assessed supervision-relevant states retain substantial gaps. Full terminal scrollback improves recovery to 73.6% but still trails full context. Under matched report lengths, context-informed rewrites improve recovery by about eight points, without a detectable targeted-over-natural advantage. Positive gaps persist across harnesses and independent DeepSeek execution and measurement. CTXGAP establishes report-based task-state recoverability as a complementary evaluation dimension.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。