arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.23675cs.LG
收藏

端到端的测试时间训练用于长上下文

End-to-End Test-Time Training for Long Context

NVIDIA(NVIDIA公司);Stanford University(斯坦福大学);UC Berkeley(伯克利大学);UC San Diego(圣地亚哥大学)
Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin,… 展开作者

Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, Jed McCaleb, Yejin Choi, Yu Sun

AI总结:

本文提出了一种端到端的测试时间训练方法,通过在测试时进行token预测和训练时的元学习,实现长上下文处理,具有与完整注意力相同的扩展性但更高效的推理速度。

摘要:

我们将长上下文语言建模视为连续学习问题,而非架构设计问题。在此框架下,我们仅使用标准架构——带有滑动窗口注意力的Transformer。然而,我们的模型通过在给定上下文中进行下一个token预测,在测试时间继续学习,将读取的上下文压缩到其权重中。此外,我们通过训练时间的元学习改进模型的初始化,以促进测试时间的学习。总体而言,我们的方法是一种测试时间训练(TTT),在测试时间(通过下一个token预测)和训练时间(通过元学习)都是端到端的,与之前的版本不同。我们进行了广泛的实验,重点在于扩展性。特别是,对于使用164B个token训练的3B模型,我们的方法(TTT-E2E)在上下文长度上的扩展方式与具有完整注意力的Transformer相同,而其他方法如Mamba 2和Gated DeltaNet则不同。然而,与RNNs类似,TTT-E2E的推理延迟与上下文长度无关,使其在128K上下文的情况下比完整注意力快2.7倍。我们的代码已公开可用。

英文摘要:

We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑