The rapid expansion of context window sizes in Large Language Models~(LLMs) has enabled them to tackle increasingly complex tasks involving lengthy documents. However, this progress comes at the cost of a substantial increase in memory usage during inference, primarily due to the linear growth of the key-value~(KV) cache. Existing KV cache compression methods often discard less relevant tokens, which can lead to significant performance degradation when critical information is lost. In this paper, we propose \textsc{PoD}~(Proximal tokens over Distant tokens), a novel KV cache compression framework that allocates memory according to token importance, retaining less important tokens in a more compact, shared form rather than discarding them entirely. Our approach is motivated by two key observations: (1) proximal tokens -- those at the beginning and end of the context -- are significantly more important for next-token prediction, and (2) attention scores for distant tokens are highly redundant across consecutive layers. Leveraging these insights, \textsc{PoD} preserves the full KV cache for proximal tokens, while for distant tokens, it shares key states across layers. Since attention scores are determined by both queries and keys, sharing key states enables multiple layers to reuse a single set of keys for distant tokens, substantially reducing KV cache memory without discarding essential context. We further introduce a lightweight post-training adaptation to enable the model to adjust to this new attention-sharing structure. Extensive experiments on both synthetic~(Needle in a Haystack) and real-world long-context benchmarks demonstrate that \textsc{PoD} reduces KV cache memory usage by up to 35\% without compromising performance. Our method is orthogonal to existing token-selection-based techniques and can be combined with them for further KV cache compression.
Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
Westlake University(西华大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(魁北克AI研究院);Zhejiang University(浙江大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德智能大学)
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Huazhong University of Science and Technology(华中科技大学);University of Electronic Science and Technology of China(电子科技大学);Carnegie Mellon University(卡内基梅隆大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);MiLM Plus, Xiaomi Inc(小米公司MiLM Plus)
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Institute of Trustworthy Embodied Artificial Intelligence, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Pengcheng Laboratory(鹏城实验室);Hikvision Inc(海康威视公司)
Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tsinghua University(清华大学);Infinigence AI;The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);SLAI;Shanghai AI Laboratory(上海人工智能实验室)
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Microsoft Research(微软研究院)
我们在MiniLM中推广了深度自我注意力蒸馏(Wang et al.,2020)通过仅使用自注意关系蒸馏来对预训练的Transformers进行任务不可知压缩。特别是,我们定义了多头自我注意力的关系,每个自我注意力模块内的查询,键和值向量对之间的缩放点积。然后运用上述相关知识对学生模型进行训练。除了简单和统一的原则,更有利的是,它没有限制的数量方面的学生的注意头,而大多数以前的工作必须保证相同的头数之间的教师和学生。此外,细粒度的自注意关系倾向于充分利用Transformer学习的交互知识。此外,我们彻底研究了教师模型的层选择策略,而不是像MiniLM中那样只依赖最后一层。我们对压缩单语和多语言预训练模型进行了广泛的实验。实验结果表明,我们的模型提取的基础大小和大尺寸的教师(BERT,RoBERTa和XLM-R)优于国家的最先进的。