智能体脚手架正越来越多地将小型决策(是否停止、验证或改变方法)委托给更廉价的模型。本研究询问仅在确定性决策点被调用的类型化决策模型Jev 1.13,是否能在不降低性能的前提下,让Claude Sonnet 5.5终端智能体的成本降低。在针对Terminal-Bench 2.1的预先指定研究中,我们固定一个智能体循环,仅改变控制器:无控制器、确定性规则、回答相同问题的小型大语言模型Claude Haiku 5.5、Jev,以及在相同时刻采样Jev开发阶段选择频率的安慰剂。在21个保留任务(315次试验)中,Jev使每次尝试的成本提升了19%(几何均值比为1.19,95%置信区间1.02-1.36),但通过率保持一致(73.0% vs. 73.0%)。Jev比小型大语言模型控制器更廉价(比值0.81),但比安慰剂贵22%,且各分支间无通过率差异。离线场景下,Jev对完成请求的排序优于其他方案(AUROC为0.78,而规则为0.62、小型大语言模型为0.60),但校准效果差:当87%的运行通过时,其认为运行完成的平均概率为0.50,因此要求智能体验证已正确的工作。没有任何控制器选择弃权(不执行)一次运行。有信息的决策信号并未成为有用的干预措施;我们讨论了实现这一点所需的条件。
英文摘要:
Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.
The Indian Financial Automated Advice Bias (IFAAB) Dataset : Auditing Socioeconomic and Demographic Disparities in LLM-Generated Financial Advice in India
LLMs Can Better Capture Human Judgments--With the Right Prompts
Complexity Science Hub, Vienna(维也纳复杂科学中心);The Ohio State University(俄亥俄州立大学);University of Cambridge(剑桥大学);University of Chicago(芝加哥大学);Microsoft Research(微软研究院)
Caught in the Story: Narrative Captivity in Multi-turn LLMs Conversation
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Dongbei University of Finance and Economics(东北财经大学)
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Michigan(密歇根大学);Carnegie Mellon University(卡内基梅隆大学);The University of Tokyo(东京大学)
Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour
York University(约克大学);Vector Institute(向量研究所);Rotman Research Institute(罗特曼研究学院);Dalhousie University(达尔豪斯大学);Centre for Addiction & Mental Health(成瘾与心理健康中心);KITE Research Institute(KITE研究机构)