arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06642cs.LGcs.AIcs.CL
收藏

SR-TTT: 基于惊奇度的残差测试时间训练

SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training

Swamynathan V P Department of Computer Science and Engineering Amrita School of Computing Amrita Vishwa Vidyapeetham Chennai, Tamil Nadu, India(斯瓦米纳坦·V·P 计算机科学与工程系 阿米塔学校 阿米塔大学 芬治, 印度泰米尔纳德邦)

Swamynathan V P

AI总结:

SR-TTT通过引入损失门控稀疏记忆机制,解决纯TTT在精确回忆任务中的失败问题,利用精确注意力处理关键信息,同时保持低内存占用。

摘要:

测试时间训练(TTT)语言模型通过将标准精确注意力KV缓存替换为隐藏状态"快速权重"W_fast,实现了理论上无限的上下文窗口,且内存占用为O(1)。然而,纯TTT架构在精确回忆任务(如针在海中找针)上会出现灾难性失败。因为快速权重会将上下文压缩成信息瓶颈,高惊奇度或独特令牌会被后续令牌的梯度更新迅速覆盖和遗忘。我们引入SR-TTT(基于惊奇度的残差测试时间训练),通过在TTT主干中添加一个损失门控稀疏记忆机制来解决这一回忆失败问题。通过动态路由仅将不可压缩、高惊奇度的令牌路由到传统精确注意力残差缓存,SR-TTT在保留O(1)内存用于低熵背景上下文的同时,利用精确注意力专门处理关键的针。我们的完整实现、训练脚本和预训练权重均已开源,可在https://github.com/swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training获取。

英文摘要:

Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks. Version 1 of this work proposed SR-TTT, which routes high-surprisal tokens to a sparse exact-attention Residual Cache, and reported large Needle-in-a-Haystack gains. We show those gains were evaluation artifacts: the loss and metric read logits at the answer positions rather than one position earlier, training both models to copy an answer already visible in their input (a model trained on retrieval-impossible data reaches 100% accuracy under the flawed metric); additionally, the cache attended non-causally over future tokens, including the answer itself. We release a corrected implementation with startup causality self-tests, then ask whether the SR-TTT hypothesis survives correction. It does not, and the failure decomposes into two independent, separately measured bottlenecks. Storage: surprisal gating is systematically position-biased - the TTT reconstruction loss requires burn-in before a needle becomes relatively surprising, so early-context needles are stored at near-zero rates (0-1% containment at depth 0.1) exactly where long-context memory matters most. Addressing: with storage solved by an oracle and with new trainable read-time projections, per-slot attention supervision raises addressing mass 2.5x (0.06 -> 0.15) yet token accuracy is statistically unchanged, and retrieval extracts only approx. 0.06 nats of the 2.30-nat needle; position-free content addressing cannot resolve ordered slots whose contents are near-interchangeable. Exact match remains 0% in all 2,250 paired trials across all corrected conditions. We retract the claims of v1 and offer the corrected codebase, diagnostic protocol, and negative results as a cautionary reference for surprise-gated memory architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑