Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. However, most of these methods treat all layers equally, allocating the same KV budget to each layer. This approach is suboptimal, as some layers may be less sensitive to input tokens yet still receive the same budget as others. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions, i.e., sequence-wise and layer-wise. Based on our observations regarding layer-wise importance in inference, we propose SqueezeAttention to precisely optimize the allocation of KV-cache budget among layers on-the-fly and then incorporate three representative sequence-wise algorithms to compress the KV-cache for each layer with its very own budget. Specifically, we first measure each layer's importance by calculating the cosine similarity of the input prompt differences before and after the self-attention layers. Based on this similarity, we then categorize the layers into two groups and adjust their KV budgets accordingly. By optimizing the KV-cache from both sequence's and layer's dimensions, SqueezeAttention achieves around 30% to 70% of the memory reductions and up to 2.2 times of throughput improvements in a wide range of LLMs and benchmarks. The code is available at https://github.com/hetailang/SqueezeAttention.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China(中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China(中国科学院计算技术研究所过程器重点实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
EvolKV: Evolutionary KV Cache Compression for LLM Inference
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CAS(复杂系统认知与决策智能重点实验室);ETH Zurich(苏黎世联邦理工学院)
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);Stepfun
ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression
Sun Yat-sen University(中山大学);The Chinese University of Hong Kong(香港中文大学);University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学);Hong Kong University(香港大学)
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Department of Computer Science, Grinnell College(格林内尔学院计算机科学系);Department of Computer Science, Rice University(莱斯大学计算机科学系);Lambda, Inc(Lambda公司);Workato(Workato公司)
SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州先进研究院,科学技术大学)
Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究所,合肥综合性国家科学中心);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州先进研究院,中国科学技术大学)
Minimal-Intervention KV Retention via Set-Conditioned Diversity
Department of Computer Science and Software Engineering(计算机科学与软件工程系);Department of Information Management(信息管理系);Auburn University(阿伯丁大学);National Central University(国立中央大学)