TurnCost:转向而非观测:面向编码智能体的缓存感知循环工程
TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents
AI总结:
该研究针对编码智能体的循环工程,发现转向数量而非观测大小是轨迹成本的主要影响因素,发布了用于评估不同策略的TurnCost工具。
摘要:
AI编码智能体在迭代循环中运行:它们修改代码、运行工具、检查结果并决定下一步操作,我们将这一过程的设计称为循环工程,其包含上下文管理、模型选择、压缩和终止。现有效率研究主要压缩工具观测结果或将选定的转向(turn)路由到更廉价的模型,但这些优化的实际机会仍不明确。我们分析了600个公开的编码智能体轨迹,涵盖两种智能体框架和两种语言模型。中位数观测结果仅包含118-286个token,而通常作为压缩动机的大型构建日志出现在第99百分位附近。此外,98.6%的输入token由前缀缓存提供,因此在接近理想缓存的情况下,固定输入定价平均高估了约7倍的循环成本。我们推导并实证验证了模型委托的闭式盈亏平衡条件,在一半的评估设置中,委托会增加成本,接近预测边界。更广泛地说,转向数量解释了轨迹对数成本方差的84%,而平均观测大小仅解释4%;最昂贵的轨迹拥有最多的转向,尽管其观测结果最小。这些关联表明,以观测为中心的方法针对的是次要成本因素,并指出不必要的转向是更大且基本未被研究的机会,不过我们并未证明在保持任务成功的同时可以移除转向。我们发布了TurnCost,这是一个用于在替代缓存、定价、压缩和路由策略下对轨迹语料库重新评分的工具。
英文摘要:
AI coding agents operate in iterative loops: they modify code, run tools, inspect the results, and decide what to do next. We call the design of this process loop engineering, which includes context management, model selection, compaction, and termination. Existing efficiency work mainly compresses tool observations or routes selected turns to cheaper models, but the practical opportunity for these optimizations remains unclear. We analyze 600 publicly released coding-agent trajectories spanning two agent scaffolds and two language models. Median observations contain only 118-286 tokens, while the large build logs commonly used to motivate compression occur near the ninety-ninth percentile. Moreover, 98.6% of input tokens are served from a prefix cache, so flat input pricing overestimates loop cost by roughly 7x on average under near-ideal caching. We derive and empirically validate a closed-form break-even condition for model delegation. Delegation increases cost in half of the evaluated settings, near the predicted boundary. More broadly, turn count explains 84% of the variance in log trajectory cost, compared with 4% for mean observation size; the most expensive trajectories have the most turns despite having the smallest observations. These associations suggest that observation-focused methods target a secondary cost factor, and identify unnecessary turns as the larger and largely unexamined opportunity, though we do not establish that turns can be removed while holding task success fixed. We release TurnCost, a tool for re-scoring trajectory corpora under alternative caching, pricing, compression, and routing policies.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。