KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine-grained decision-making, and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of layer-wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We further propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference. To reduce the computational cost of offline calibration, we utilize the intra-layer KV precision pair pruning and inter-layer clustering to reduce the search space. Experimental results show that we can achieve nearly lossless 3.25-bit mixed precision KV cache quantization for LLMs like Llama-3.1-8B-Instruct and 4.0-bit for sensitive models like Qwen2.5-7B-Instruct on mathematical reasoning tasks. The maximum inference throughput can be improved by 21.25\% compared with KIVI-KV8 quantization over various context lengths. Our code and searched configurations are available at https://github.com/cmd2001/KVTuner.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China（中国科学技术大学）；State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China（中国科学院计算技术研究所过程器重点实验室，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
Westlake University（西华大学）；McGill University（麦吉尔大学）；Mila - Quebec AI Institute（魁北克AI研究院）；Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德智能大学）
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；MiLM Plus, Xiaomi Inc（小米公司MiLM Plus）
Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
MoE Key Laboratory of Interdisciplinary Research of Computation and Economics（交叉计算与经济学 interdisciplinary 研究联合实验室）；Shanghai University of Finance and Economics（上海财经大学）
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；UCAS（中国科学技术大学）；The Hong Kong Polytechnic University（香港理工大学）
Microsoft（微软）；University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and prevents leveraging larger batch size for faster speed. We propose memory-efficient lossless attention (called EL-attention) to address this issue. It avoids heavy operations for building multi-head keys and values, cache for them is not needed. EL-attention constructs an ensemble of attention results by expanding query while keeping key and value shared. It produces the same result as multi-head attention with less GP…
ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
School of Electronic Science and Engineering, Nanjing University, China.（南京大学电子科学与工程学院）；National Key Laboratory for Novel Software Technology, Nanjing University, China.（南京大学新型软件技术国家重点实验室）
UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective
The University of Hong Kong（香港大学）；Nanjing University（南京大学）；University College London（伦敦大学学院）；The Ohio State University（俄亥俄州立大学）；Microsoft Research Asia（微软亚洲研究院）；The Chinese University of Hong Kong（香港中文大学）；SIAT, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Princeton University（普林斯顿大学）；Together AI；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Connecticut（康涅狄格大学）
