The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Chinese Academy of Sciences(中国科学院大学);CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院人工智能安全重点实验室);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology(哈尔滨工业大学);University of Oxford(牛津大学);Tsinghua University(清华大学)
Ao Sun, Hongtao Zhang, Heng Zhou, Yixuan Ma, Yiran Qin, Tongrui Su, Yan Liu, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He
Linear Attention Large Language Models (LLMs) offer a compelling recurrent formulation that compresses context into a fixed-size state matrix, enabling constant-time inference. However, the internal dynamics of this compressed state remain largely opaque. In this work, we present a comprehensive study on the runtime state dynamics of state-of-the-art Linear Attention models. We uncover a fundamental phenomenon termed State Rank Stratification, characterized by a distinct spectral bifurcation among linear attention heads: while one group maintains an effective rank oscillating near zero, the other exhibits rapid growth that converges to an upper bound. Extensive experiments across diverse inference contexts reveal that these dynamics remain strikingly consistent, indicating that the identity of a head,whether low-rank or high-rank,is an intrinsic structural property acquired during pre-training, rather than a transient state dependent on the input data. Furthermore, our diagnostic probes reveal a surprising functional divergence: low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy. Leveraging this insight, we propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9\% reduction in KV-cache overhead while largely maintaining model accuracy.
Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Beijing Institute of Technology(北京理工大学);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);MMLab, The Chinese University of Hong Kong(香港中文大学MMLab);Hebei Province Key Laboratory of Big Data Science and Intelligent Technology(河北省大数据科学与智能技术重点实验室)
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通大数据挖掘与具身智能重点实验室);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);State Key Laboratory of Advanced Rail Autonomous Operation(先进轨道交通自主运行国家重点实验室)
Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs
Theoretical Division, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室理论分部);Advanced Research in Cyber Systems, Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室网络系统高级研究分部);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
Enabling Weak LLMs to Judge Response Reliability via Meta Ranking
Tsinghua University(清华大学);Weiyang College, Tsinghua University(清华大学未央书院);Institute of Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究所);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能国际研究院)
Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);Data61;CSIRO(联邦科学与工业研究组织);Tsinghua University(清华大学);Australian National University(澳大利亚国立大学)
SLASH the Sink: Sharpening Structural Attention Inside LLMs
Shanghai Jiao Tong University(上海交通大学);Institute of Geographical Science and Natural Resources Research(地理科学与自然资源研究所);Chinese Academy of Sciences(中国科学院)
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs
Beijing University of Posts and Telecommunications(北京邮电大学);Hangzhou Dianzi University(杭州电子科技大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Fujian Cancer Hospital(福建癌症医院)
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Lehigh University(莱维大学);University of Notre Dame(诺特大学);Zhejiang Wanli University(浙江万里大学);Squirrel Ai Learning;City University of Hong Kong(香港城市大学);Duke University(杜克大学)
School of Mathematics, Northwest University(西北大学数学学院);Shanghai Institute for Mathematics and Interdisciplinary Sciences(上海数学与交叉科学研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Cheriton School of Computer Science, University of Waterloo(滑铁卢大学切尔顿计算机科学学院)