The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.
VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；MiLM Plus, Xiaomi Inc（小米公司MiLM Plus）
Accurate KV Cache Quantization with Outlier Tokens Tracing
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室）；Huawei Cloud（华为云）
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
LUMIA Lab（LUMIA实验室）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Cambridge（剑桥大学）；National University of Singapore（新加坡国立大学）
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）；Xiaomi Inc.（小米公司）
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
School of Information Technology and Management, University of International Business and Economics（对外经济贸易大学信息技术与管理学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；School of Information, Renmin University of China（中国人民大学信息学院）
AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
DSA Thrust, The Hong Kong University of Science and Technology (Guangzhou)（DSA团队，香港科学与技术大学（广州））；School of Mathematics and Computational Science, Xiangtan University（数学与计算科学学院，湘潭大学）；Microsoft（微软）
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；UCAS（中国科学技术大学）；The Hong Kong Polytechnic University（香港理工大学）
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；MiLM Plus, Xiaomi Inc（小米公司MiLM Plus）
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA（计算机科学与软件工程系，阿伯拉罕大学，阿伯丁，阿拉巴马州，美国）；Department of Information Management, National Central University, Taoyuan, Taiwan（信息管理系，国立中央大学，桃园，台湾）
