VL-Cache: 面向视觉-语言模型推理加速的稀疏性与模态感知KV缓存压缩
VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
AI总结：
针对现有KV缓存压缩方法直接迁移到视觉-语言模型效果不佳的问题，提出兼具稀疏性与模态感知的VL-Cache压缩方案，在仅保留10%缓存的情况下保持精度，实现显著的推理加速与内存节省。
摘要：
视觉-语言模型（Vision-Language Models, VLMs）已在多种任务中展现出优异性能。加速VLMs的一大核心挑战是存储与访问编码了图像、视频等长视觉上下文的大型键值（Key-Value, KV）缓存。现有KV缓存压缩方法虽对大语言模型（Large Language Models, LLMs）有效，但直接迁移到VLMs会导致精度与加速效果均不理想。为弥补这一差距，我们提出VL-Cache——一种专为加速VLM推理设计的新型KV缓存压缩方案。本文首先通过区分预填充与解码阶段的视觉token和文本token，探究了VLM注意力的独特稀疏模式。基于这些观察，我们提出了一种层自适应稀疏感知缓存预算分配方法，能将有限的缓存预算有效分配到不同层，在不损失精度的前提下进一步缩小KV缓存规模。此外，我们还设计了模态感知的token打分策略，以更准确地评估token的重要性。在多个基准数据集上的实验结果表明，仅保留10%的KV缓存即可达到与全缓存相当的精度。在速度基准测试中，我们的方法将生成100个token的端到端延迟最高加速2.33倍，解码速度最高提升7.08倍，同时将GPU中KV缓存的内存占用降低了90%。
英文摘要：
Vision-Language Models (VLMs) have demonstrated impressive performance across a versatile set of tasks. A key challenge in accelerating VLMs is storing and accessing the large Key-Value (KV) cache that encodes long visual contexts, such as images or videos. While existing KV cache compression methods are effective for Large Language Models (LLMs), directly migrating them to VLMs yields suboptimal accuracy and speedup. To bridge the gap, we propose VL-Cache, a novel KV cache compression recipe tailored for accelerating VLM inference. In this paper, we first investigate the unique sparsity pattern of VLM attention by distinguishing visual and text tokens in prefill and decoding phases. Based on these observations, we introduce a layer-adaptive sparsity-aware cache budget allocation method that effectively distributes the limited cache budget across different layers, further reducing KV cache size without compromising accuracy. Additionally, we develop a modality-aware token scoring policy to better evaluate the token importance. Empirical results on multiple benchmark datasets demonstrate that retaining only 10% of KV cache achieves accuracy comparable to that with full cache. In a speed benchmark, our method accelerates end-to-end latency of generating 100 tokens by up to 2.33x and speeds up decoding by up to 7.08x, while reducing the memory footprint of KV cache in GPU by 90%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
