arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.04852cs.LG
收藏

线性注意力的状态缩减:基于秩的视角

On State Reduction in Linear Attention

ETH Zürich(苏黎世联邦理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心)

Philipp Nazari, T. Konstantin Rusch

AI总结:

本文提出了一种基于秩的线性注意力状态缩减方法,通过结构化修剪减少状态大小,同时保持模型性能,实现更快和更节省内存的模型。

摘要:

线性注意力提供了一种计算高效且表达能力强的替代方案,相较于softmax注意力。然而,最近的实证结果表明,训练后的线性注意力模型的隐藏状态往往表现出低秩结构,这表明这些模型在实践中未能充分利用其容量。为了阐明这一现象,我们对线性注意力中秩的作用进行了理论分析,揭示低有效秩会影响检索误差,通过放大查询噪声。除了这些理论见解外,我们推测在训练后通过少量性能损失即可大幅减少低秩状态,从而得到更快且更节省内存的模型。为此,我们提出了一种新型的硬件感知方法,通过结构化修剪键和查询矩阵,减少状态大小的同时保持与现有CUDA内核的兼容性。我们适应了几种现有的修剪策略以适应我们的框架,并基于我们的理论分析,提出了一种基于秩揭示QR分解的新型结构化修剪方法。我们的实证结果,在不同规模的模型和各种下游任务上进行了评估,展示了我们状态缩减框架的有效性。我们强调,我们的框架能够仅在词法复杂度上略微增加的情况下,移除50%的查询和键通道。该项目的代码可在https://github.com/camail-official/LinearAttentionPruning中找到。

英文摘要:

Linear attention offers a computationally efficient yet expressive alternative to softmax attention. However, recent empirical results indicate that the hidden state of trained linear attention models often exhibits a low-rank structure, suggesting that these models underexploit their capacity in practice. To understand this phenomenon, we analyze how keys and values shape the rank of the recurrent state, providing a theoretical perspective on memory utilization in linear attention. In addition to these theoretical insights, we conjecture that low-rank states can be substantially reduced after pre-training. To this end, we propose a hardware-aware strategy to structurally prune the key and query matrices, reducing the state size while retaining compatibility with existing fast kernels. We adapt several existing pruning strategies to fit our framework and, building on our theoretical analysis, propose a robust structured pruning method based on a rank-revealing QR decomposition. Our empirical evaluations across model sizes show that the recurrent state of pretrained linear attention models can often be halved with only a modest increase in language modeling perplexity, while maintaining competitive performance on downstream tasks. The code for this project can be found at https://github.com/camail-official/LinearAttentionPruning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑