arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02195cs.LGcs.AI
收藏

线性注意力大语言模型中的状态排名动态

State Rank Dynamics in Linear Attention LLMs

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Chinese Academy of Sciences(中国科学院大学);CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院人工智能安全重点实验室);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology(哈尔滨工业大学);University of Oxford(牛津大学);Tsinghua University(清华大学)

Ao Sun, Hongtao Zhang, Heng Zhou, Yixuan Ma, Yiran Qin, Tongrui Su, Yan Liu, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

AI总结:

本研究揭示了线性注意力模型中状态排名分层现象,提出联合秩-范数剪枝策略,有效降低KV缓存开销并维持模型准确性。

摘要:

线性注意力大语言模型(LLMs)提供了一种具有吸引力的递归公式,能够将上下文压缩成固定大小的状态矩阵,从而实现常数时间推断。然而,这种压缩状态的内部动态仍然大多模糊不清。在本工作中,我们对最先进的线性注意力模型的运行时状态动态进行了全面研究。我们揭示了一个基本现象,称为状态排名分层,其特征是线性注意力头之间存在明显的谱分岔:一组头保持有效的秩在零附近震荡,而另一组则表现出快速增长,最终收敛到一个上界。在各种推理上下文中进行的广泛实验表明,这些动态在不同情况下仍然令人惊讶地一致,表明头的身份,无论是低秩还是高秩,都是在预训练过程中获得的内在结构属性,而不是依赖于输入数据的瞬时状态。此外,我们的诊断探针揭示了令人惊讶的功能差异:低秩头对于模型推理是必不可少的,而高秩头则表现出显著的冗余性。利用这一见解,我们提出了联合秩-范数剪枝,这是一种零样本策略,能够在减少KV缓存开销38.9%的同时,大体上保持模型的准确性。

英文摘要:

Linear Attention Large Language Models (LLMs) offer a compelling recurrent formulation that compresses context into a fixed-size state matrix, enabling constant-time inference. However, the internal dynamics of this compressed state remain largely opaque. In this work, we present a comprehensive study on the runtime state dynamics of state-of-the-art Linear Attention models. We uncover a fundamental phenomenon termed State Rank Stratification, characterized by a distinct spectral bifurcation among linear attention heads: while one group maintains an effective rank oscillating near zero, the other exhibits rapid growth that converges to an upper bound. Extensive experiments across diverse inference contexts reveal that these dynamics remain strikingly consistent, indicating that the identity of a head,whether low-rank or high-rank,is an intrinsic structural property acquired during pre-training, rather than a transient state dependent on the input data. Furthermore, our diagnostic probes reveal a surprising functional divergence: low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy. Leveraging this insight, we propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9\% reduction in KV-cache overhead while largely maintaining model accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑