建议的变更未被取代:对作为记忆退役门的系统一模型的预注册评估
Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate
AI总结:
该研究针对助手长期记忆的记忆退役门控,通过预注册实验评估Jev模型,发现二元门控过早退役建议变更的问题,提出三分决策方案并指出级联模型的局限性,建议将建议变更路由至待处理证据。
摘要:
助手的长期记忆需要一个门控机制,用于决定何时新话语会使旧存储的声明退役。该步骤的基准标签将一对标记为已变更或未变更,因此无法区分基于已确立变更的门控与基于提示的门控。我们编写了一个任务契约,将两者分开(取代 vs. 需要确认 vs. 兼容),使用三个大语言模型(LLM)标注员和两名人工仲裁员(一名作者和第二名标注员;标注来源已报告)对从STALE(变更对加同用户对照)构建的500个测试对进行标注,并在预注册假设下评估Jev,这是一个类型化的“系统一”决策模型。在STALE标记为变更的179个测试对中,有72个达到了契约对已确立变更的标准。Jev从未对141个同对话非变更的声明进行退役。其失败在于过早退役:二元门控会使39.2%仅为建议变更的声明退役。三分决策将该比例降至15.5%,但检测成本上升了15.3个百分点,超出了我们预注册的10个百分点预算,且结果处于二元门控自身的阈值前沿;这些结论在对黄金标签的三种解读下均成立。在针对第二名标注员的标签评分的保留库上,一个将不确定案例升级至推理模型且在开发数据上表现最佳的级联模型,与单纯提高阈值的效果无差异(开发数据上过早退役比例分别为26.8% vs. 6.5%)。部分差距源于黄金标签:相对于该标注员的解读,LLM多数黄金标签的大部分错误与同系列的升级模型共享,而二元标签的一致性表明此类共享错误隐藏了多少信息。其余差距源于开发数据上的小样本选择。对于构建者而言,这意味着不应将二元分数与删除操作绑定:应将建议变更路由至待处理证据,且不要信任基于此规模开发数据选择的级联模型。
英文摘要:
Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair as changed or not, so they cannot tell a gate that acts on established change from one that acts on a hint. We write a task contract that separates the two (SUPERSEDED vs. NEEDS-CONFIRMATION vs. COMPATIBLE), label 500 test pairs built from STALE (change pairs plus same-user controls) with three LLM annotators and two human adjudicators (one author and a second annotator; provenance reported), and evaluate Jev, a typed "System One" decision model, under pre-registered hypotheses. Of the 179 test pairs STALE marks as changed, 72 meet the contract's bar for an established change. Jev never retired a claim on 141 same-conversation non-changes. Its failure is premature retirement: the binary gate retires 39.2% of claims whose change is only suggested. A three-way decision cuts this to 15.5% at a 15.3-point detection cost, which misses our pre-registered 10-point budget, and it lands on the binary gate's own threshold frontier; these verdicts hold under three readings of the gold labels. On a held-out vault scored against a second annotator's labels, a cascade that escalated uncertain cases to a reasoning model and looked best on development data was indistinguishable from simply raising the threshold (premature retirement 26.8% vs. 6.5% on dev). Part of the gap comes from the gold labels: relative to that annotator's reading, LLM-majority gold shares most of its errors with a same-family escalation model, and an identity for binary labels says how much such shared errors hide. The rest is small-sample selection on development data. For builders this means a binary score should not be wired to delete: route suggested changes to pending evidence, and do not trust a cascade selected on development data of this size.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。