The advent of pre-trained large language models (LLMs) has revolutionized various natural language processing tasks. These models predominantly employ an auto-regressive decoding mechanism that utilizes Key-Value (KV) caches to eliminate redundant calculations for previous tokens. Nevertheless, as context lengths and batch sizes increase, the linear expansion in memory footprint of KV caches becomes a key bottleneck of LLM deployment, which decreases generation speeds significantly. To mitigate this issue, previous techniques like multi-query attention (MQA) and grouped-query attention (GQA) have been developed, in order to reduce KV heads to accelerate inference with comparable accuracy to multi-head attention (MHA). Despite their effectiveness, existing strategies for compressing MHA often overlook the intrinsic properties of the KV caches. In this work, we explore the low-rank characteristics of the KV caches and propose a novel approach for compressing KV heads. In particular, we carefully optimize the MHA-to-GQA transformation to minimize compression error, and to remain compatible with rotary position embeddings (RoPE), we also introduce specialized strategies for key caches with RoPE. We demonstrate that our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs, which presents a promising direction for more efficient LLM deployment in resource-constrained environments.
Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
Westlake University(西华大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(魁北克AI研究院);Zhejiang University(浙江大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德智能大学)
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
School of Software Technology, Zhejiang University(浙江大学软件学院);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);Geely Automobile Research Institute (Ningbo) Co., Ltd.(吉利汽车研究院(宁波)有限公司);FABU Inc.(FABU公司)
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China(中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China(中国科学院计算技术研究所过程器重点实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
LongHeads: Multi-Head Attention is Secretly a Long Context Processor
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);International Human Phenome Institutes, Shanghai, China(国际人类表型组研究院(上海))
Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AI Lab, The Yangtze River Delta(长江三角洲人工智能实验室);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University College London(伦敦大学学院);The Hong Kong University of Science and Technology(香港科学与技术大学);UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心)
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Institute of Trustworthy Embodied Artificial Intelligence, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Pengcheng Laboratory(鹏城实验室);Hikvision Inc(海康威视公司)
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Microsoft Research(微软研究院)
我们在MiniLM中推广了深度自我注意力蒸馏(Wang et al.,2020)通过仅使用自注意关系蒸馏来对预训练的Transformers进行任务不可知压缩。特别是,我们定义了多头自我注意力的关系,每个自我注意力模块内的查询,键和值向量对之间的缩放点积。然后运用上述相关知识对学生模型进行训练。除了简单和统一的原则,更有利的是,它没有限制的数量方面的学生的注意头,而大多数以前的工作必须保证相同的头数之间的教师和学生。此外,细粒度的自注意关系倾向于充分利用Transformer学习的交互知识。此外,我们彻底研究了教师模型的层选择策略,而不是像MiniLM中那样只依赖最后一层。我们对压缩单语和多语言预训练模型进行了广泛的实验。实验结果表明,我们的模型提取的基础大小和大尺寸的教师(BERT,RoBERTa和XLM-R)优于国家的最先进的。
Steering at the Source: Style Modulation Heads for Robust Persona Control
The University of Tokyo, Tokyo, Japan(东京大学);National Institute of Informatics Research(信息处理研究所);Development Center for Large Language Models, Japan(大型语言模型发展中心)
Safety Alignment Should Be Made More Than Just A Few Attention Heads
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);DAIL Tech(DAIL科技);Northeastern University(东北大学);Sichuan University(四川大学);Huazhong University of Science and Technology(华中科技大学)
Attention Head Masking for Inference Time Content Selection in Abstractive Summarization
University of Michigan(密歇根大学)
如何在基于transformer的抽象摘要模型中有效地通知内容选择?在这项工作中,我们提出了一个简单而有效的注意头掩蔽技术,这是适用于编码器-解码器的注意力,以查明在推理时间的显着内容。利用注意力头部掩蔽,我们能够揭示编码器-解码器注意力和摘要模型的内容选择行为之间的关系。然后,我们证明了其有效性的三个文档摘要数据集的基础上,在域和跨域设置。重要的是,我们的模型在CNN/Daily Mail和New York Times数据集上的表现优于之前最先进的模型。此外,我们的推理时间掩蔽技术也是数据高效的,只需要20%的训练样本就可以在完整的CNN/DailyMail数据集上超越BART微调。
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国);School of Software, Soongsil University, Seoul, South Korea(软件学院,顺天大学,首尔,韩国)