arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21204cs.LGcs.AIcs.CV
收藏

测试时训练与KV绑定实际上是秘密的线性注意力

Test-Time Training with KV Binding Is Secretly Linear Attention

NVIDIA, Toronto, Ontario, Canada(NVIDIA,多伦多,安大略省,加拿大);University of Toronto, Toronto, Ontario, Canada(多伦多大学,多伦多,安大略省,加拿大);Vector Institute, Toronto, Ontario, Canada(向量研究所,多伦多,安大略省,加拿大);Technion -- Israel Institute of Technology, Haifa, Israel(技术ion -- 以色列理工学院,海法,以色列)

Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li

AI总结:

本文揭示测试时训练与KV绑定实际上是学习的线性注意力,而非记忆,提出简化架构、并行计算和统一模型形式的改进方法。

摘要:

测试时训练(TTT)与KV绑定作为序列建模层通常被解释为一种在线元学习,通过在测试时记忆键值映射。然而,我们的分析揭示了多个现象,这些现象与基于记忆的解释相矛盾。受这些发现的启发,我们重新审视TTT的公式化,证明了一类广泛的TTT架构可以表示为学习的线性注意力操作符。除了解释之前令人困惑的模型行为外,这种视角还带来了多个实际益处:它使架构简化成为原则性,允许完全并行的公式化,从而在保持性能的同时提高效率,并将各种TTT变体系统地减少到标准线性注意力形式。总体而言,我们的结果将TTT重新定义为学习的线性注意力,而非测试时的记忆,具有增强的表示能力。项目页面:https://research.nvidia.com/labs/sil/projects/tttla/.

英文摘要:

Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑