AirCache：激活模态间相关性的KV缓存压缩，以实现高效的大型视觉-语言模型推理
AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference
AI总结：
本文提出AirCache，通过模态间相关性评估视觉KV缓存重要性并自适应分配逐层预算，在仅保留10%视觉缓存时保持性能，并将解码延迟降低29%至66%。
摘要：
近期，大型视觉语言模型（Large Visual Language Models，LVLMs）的进展因其出色的推理能力和泛化熟练度而获得了广泛关注。然而，处理大量视觉token并生成长上下文输出会带来巨大的计算开销，导致对键值（key-value，KV）缓存的过度需求。为解决这一关键瓶颈，我们提出了AirCache，这是一种旨在加速LVLMs推理的新型KV缓存压缩方法。本工作系统研究了LVLMs注意力机制中视觉token与文本token之间的相关性。我们的实证分析揭示了缓存视觉token中存在相当大的冗余；在这种情况下，有策略地剔除这些token可以保持模型性能，同时显著加速上下文生成。受这些发现启发，我们引入了一个精英观察窗口，用于评估KV缓存中视觉成分的重要性，重点关注稳定的模态间相关性建模，并增强多视角一致性。此外，我们开发了一种自适应的逐层预算分配策略，该策略利用token重要性分布的强度和偏度，相比均匀分配展现出更优的效率。跨多个LVLMs和基准的综合评估表明，我们的方法在仅保留10%视觉KV缓存的情况下，取得了与完整缓存相当的性能，从而在不同批大小和输入提示长度下将解码延迟降低29%至66%。值得注意的是，随着缓存保留率下降，我们的方法相比现有方法表现出不断增大的性能优势。
英文摘要：
Recent advancements in Large Visual Language Models (LVLMs) have gained significant attention due to their remarkable reasoning capabilities and proficiency in generalization. However, processing a large number of visual tokens and generating long-context outputs impose substantial computational overhead, leading to excessive demands for key-value (KV) cache. To address this critical bottleneck, we propose AirCache, a novel KV cache compression method aimed at accelerating LVLMs inference. This work systematically investigates the correlations between visual and textual tokens within the attention mechanisms of LVLMs. Our empirical analysis reveals considerable redundancy in cached visual tokens, wherein strategically eliminating these tokens preserves model performance while significantly accelerating context generation. Inspired by these findings, we introduce an elite observation window for assessing the importance of visual components in the KV cache, focusing on stable inter-modal relevancy modeling with enhanced multi-perspective consistency. Additionally, we develop an adaptive layer-wise budget allocation strategy that capitalizes on the strength and skewness of token importance distribution, showcasing superior efficiency compared to uniform allocation. Comprehensive evaluations across multiple LVLMs and benchmarks demonstrate that our method achieves comparable performance to the full cache while retaining only 10% of visual KV cache, thereby reducing decoding latency by 29% to 66% across various batch size and prompt length of inputs. Notably, as cache retention rates decrease, our method exhibits increasing performance advantages over existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
