A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation. However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation. In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective, significantly reducing the memory footprint for LLM inference. Our approach is based on the observation that KV cache states exhibit high similarity between the adjacent layers in the middle-to-deep portion of LLMs. To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead. Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity. We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances inference throughput by approximately 5x, and reduces the memory footprint by 41% compared to the FP16 full cache baseline, all while maintaining near-lossless performance.
SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
Centre for Responsible AI（负责任人工智能中心）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Department of Electrical Engineering（电气工程系）；Wadhwani School of Data Science and Artificial Intelligence（Wadhwani数据科学与人工智能学院）
R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Microsoft（微软）；Carnegie Mellon University（卡内基梅隆大学）；California Institute of Technology（加州理工学院）；University of California - San Diego（加州大学圣地亚哥分校）；University of Surrey（萨里大学）；University of California - Berkeley（加州大学伯克利分校）
EvolKV: Evolutionary KV Cache Compression for LLM Inference
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CAS（复杂系统认知与决策智能重点实验室）；ETH Zurich（苏黎世联邦理工学院）
Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
The University of Chicago（芝加哥大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at UC Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）
AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知联合实验室，中国科学技术大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学）
ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学信息中心）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
