arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.24649cs.AIcs.LG
收藏

良好判断,糟糕建议:决策模型控制器在不提升成功率的情况下增加智能体成本

Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

Naman Vats

AI总结:

本研究在Terminal-Bench 2.1上测试Jev 1.13等控制器对Claude Sonnet 5.5智能体的影响,发现Jev虽排序完成请求表现较好,但会增加19%成本且不提升通过率。

摘要:

智能体脚手架正越来越多地将小型决策(是否停止、验证或改变方法)委托给更廉价的模型。本研究询问仅在确定性决策点被调用的类型化决策模型Jev 1.13,是否能在不降低性能的前提下,让Claude Sonnet 5.5终端智能体的成本降低。在针对Terminal-Bench 2.1的预先指定研究中,我们固定一个智能体循环,仅改变控制器:无控制器、确定性规则、回答相同问题的小型大语言模型Claude Haiku 5.5、Jev,以及在相同时刻采样Jev开发阶段选择频率的安慰剂。在21个保留任务(315次试验)中,Jev使每次尝试的成本提升了19%(几何均值比为1.19,95%置信区间1.02-1.36),但通过率保持一致(73.0% vs. 73.0%)。Jev比小型大语言模型控制器更廉价(比值0.81),但比安慰剂贵22%,且各分支间无通过率差异。离线场景下,Jev对完成请求的排序优于其他方案(AUROC为0.78,而规则为0.62、小型大语言模型为0.60),但校准效果差:当87%的运行通过时,其认为运行完成的平均概率为0.50,因此要求智能体验证已正确的工作。没有任何控制器选择弃权(不执行)一次运行。有信息的决策信号并未成为有用的干预措施;我们讨论了实现这一点所需的条件。

英文摘要:

Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑