The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token's future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors. Each agent learns a specialized eviction policy guided by a holistic reward, derived from future utility, that evaluates the quality of the ranking across all cache budgets, requiring no modifications to the underlying LLM or additional inference. Evaluated across two model families on the long-context benchmark RULER (up to 128K tokens) and the multi-turn dialogue benchmark OASST2-4k, KVP significantly outperforms strong baselines. Zero-shot tests on standard downstream tasks (BoolQ, LongBench passage retrieval, GovReport) further show that KVP generalizes beyond its training distribution and to considerably longer sequence lengths. These results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China(中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China(中国科学院计算技术研究所过程器重点实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Baidu Inc.(百度公司);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Department of Computer Science, Grinnell College(格林内尔学院计算机科学系);Department of Computer Science, Rice University(莱斯大学计算机科学系);Lambda, Inc(Lambda公司);Workato(Workato公司)
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Huazhong University of Science and Technology(华中科技大学);University of Electronic Science and Technology of China(电子科技大学);Carnegie Mellon University(卡内基梅隆大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
University of Electronic Science and Technology of China(电子科技大学);Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province(四川省泛在智能与可信服务重点实验室)
Taming the Fragility of KV Cache Eviction in LLM Inference
University of Science and Technology of China(中国科学技术大学);USTC;Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research(数据暗夜实验室、MIRACLE中心、苏州先进研究所)
G-KV: Decoding-Time KV Cache Eviction with Global Attention
School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);MicroSoft(微软);Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence(北京交通数据挖掘与具身智能重点实验室)
SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州先进研究院,科学技术大学)
CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
School of Computer Science, University of Science and Technology of China(科学技术大学计算机科学学院);School of Biomedical Engineering, USTC(USTC生物医学工程学院);Data Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research(苏州先进研究院数据黑暗实验室、奇迹中心);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院)
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
College of Computer Science, Sichuan University, Chengdu, China(四川大学计算机学院,成都,中国);Institute of High Performance Computing, A STAR, Singapore(高性能计算研究院,新加坡)
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(基础模型研究中心,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);University of Electronic Science and Technology of China(电子科技大学);Wuhan AI Research(武汉人工智能研究);Objecteye Inc.(Objecteye公司)