The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system. This paper enables a novel dynamic KV-Cache eviction policy by injecting a lightweight module called Attention-Gate to the model. It accepts the global context as input and yields eviction flags for each token. The self-attention modules in the model proceed according to the flags and cache only a subset of the KV states for next token prediction. The Attention-Gates can yield various flags for different heads and layers and be easily tuned on top of a pre-trained LLM via continual pre-training or supervised fine-tuning. The computational and memory overhead introduced by Attention-Gates can be minimal. We empirically evaluate the proposed approach across multiple scenarios, showing that effective eviction of redundant tokens can not only improve efficiency but also enhance performance.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China（中国科学技术大学）；State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China（中国科学院计算技术研究所过程器重点实验室，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Baidu Inc.（百度公司）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
University of Electronic Science and Technology of China（电子科技大学）；Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province（四川省泛在智能与可信服务重点实验室）
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Department of Computer Science, Grinnell College（格林内尔学院计算机科学系）；Department of Computer Science, Rice University（莱斯大学计算机科学系）；Lambda, Inc（Lambda公司）；Workato（Workato公司）
G-KV: Decoding-Time KV Cache Eviction with Global Attention
School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）；MicroSoft（微软）；Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence（北京交通数据挖掘与具身智能重点实验室）
SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州先进研究院，科学技术大学）
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Huazhong University of Science and Technology（华中科技大学）；University of Electronic Science and Technology of China（电子科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Taming the Fragility of KV Cache Eviction in LLM Inference
University of Science and Technology of China（中国科学技术大学）；USTC；Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research（数据暗夜实验室、MIRACLE中心、苏州先进研究所）
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院，成都，中国）；Institute of High Performance Computing, A STAR, Singapore（高性能计算研究院，新加坡）
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
School of Computer Science, University of Science and Technology of China（科学技术大学计算机科学学院）；School of Biomedical Engineering, USTC（USTC生物医学工程学院）；Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research（苏州先进研究院数据黑暗实验室、奇迹中心）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（基础模型研究中心，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；University of Electronic Science and Technology of China（电子科技大学）；Wuhan AI Research（武汉人工智能研究）；Objecteye Inc.（Objecteye公司）
