The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed KV vectors. However, its memory consumption scales linearly with sequence length and batch size, posing a significant bottleneck in LLM deployment. Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific.
This paper introduces an orthogonal approach to KV cache compression. We propose a low-rank approximation of KV weight matrices, allowing for plug-in integration with existing transformer-based LLMs without model retraining. To effectively compress KV cache at the weight level, we adjust for layerwise sensitivity and introduce a progressive compression strategy, which is supported by our theoretical analysis on how compression errors accumulate in deep networks. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages. Extensive experiments with LLaMA models ranging from 8B to 70B parameters across various tasks show that our approach significantly reduces the GPU memory footprint while maintaining performance.
CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery
School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学科学学院）；School of Mathematics and Statistics, Chongqing University（重庆大学数学与统计学院）；School of Science, Beijing Forestry University（北京林业大学理学院）
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Peking University（北京大学）；Nanjing University（南京大学）；University of Surrey（Surrey 大学）；Qwen（通义实验室）；University of California - Riverside（加州大学河滨分校）；Microsoft（微软公司）
Integrative Sciences and Engineering Programme, National University of Singapore（国立新加坡大学整合科学与工程学程）；School of Computing, National University of Singapore（国立新加坡大学计算机学院）
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization
University of California, Berkeley（加州大学伯克利分校）；NVIDIA；Massachusetts Institute of Technology（麻省理工学院）；Amazon（亚马逊）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
University of Science and Technology of China（中国科学技术大学）；Shanghai Advanced Research Institute, Chinese Academy of Sciences（上海先进研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
MemoVAD: Resource-Efficient Video Anomaly Detection via Dynamic Semantic Memory in Edge Computing Scenarios
Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Engineering Research Center of Cloud-Edge Intelligent Collaboration on Big Data, Ministry of Education, Beijing Normal University（北京师范大学大数据云边智能协同教育部工程研究中心）
ConCise: Training-Free Conclusion-Chain State Compression for Cost-Efficient Multi-Step RAG Services
National Natural Science Foundation of China（国家自然科学基金委员会）；Guangdong Higher Education Association（广东省高等教育协会）；Guangdong Provincial Higher Education Institutions（广东省高等教育机构）；Beijing Normal University at Zhuhai Education Reform Project（珠海市北京师范大学教育改革项目）
