arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.00671cs.CLcs.AI
收藏

快速权重产品键内存

Fast-weight Product Key Memory

Sakana AI

Tianyu Zhao, Llion Jones

AI总结:

FwPKM是一种稀疏快速权重内存层,通过稀疏梯度下降实现高效存储与检索,有效降低困惑度并提升长上下文处理能力。

摘要:

现代语言模型中的序列建模层通常面临存储容量与计算效率之间的权衡。虽然softmax注意力提供了无限制的存储能力,但代价是二次的高昂成本,而线性变体则更高效,但存在有限且固定大小的存储限制。我们引入了快速权重产品键内存(FwPKM),这是一种稀疏快速权重内存层,解决了这一矛盾。FwPKM在训练和推理过程中通过在局部内存重写目标上进行基于块的梯度下降,稀疏地更新激活参数。这使FwPKM能够在稀疏内存中执行类似测试时间训练(TTT)风格的梯度更新,从而在保持每token计算低且固定的情况下,快速记忆和检索许多新的键值关联。实验表明,FwPKM作为一种有效的事件性记忆,能够补充标准模块的语义记忆,从而在长上下文数据集上显著降低困惑度。值得注意的是,在针尖在海草中的评估中,尽管仅在4K-token序列上进行训练,FwPKM仍能泛化到128K-token的上下文中。

英文摘要:

Sequence modeling layers in modern language models typically face a trade-off between storage capacity and computational efficiency. While softmax attention offers unbounded storage at prohibitive quadratic cost, linear variants are more efficient but suffer from limited, fixed-size storage. We introduce Fast-weight Product Key Memory (FwPKM), a sparse fast-weight memory layer that resolves this tension. FwPKM updates sparsely activated parameters at both training and inference time using chunk-level gradient descent on a local memory-rewrite objective. This performs Test-Time Training (TTT)-style gradient updates on activated slots in a sparse memory, enabling rapid memorization and retrieval of many new key-value associations while keeping per-token compute low and fixed. Experiments show that FwPKM functions as an effective episodic memory that complements the semantic memory of standard modules, yielding significant perplexity reductions on long-context datasets. Notably, in Needle-in-a-Haystack evaluations, FwPKM generalizes to 128K-token contexts despite being trained on only 4K-token sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑