使大型视觉-语言模型的KV缓存更轻量
Make Your LVLM KV Cache More Lightweight
AI总结：
本文提出LightKV方法，通过利用视觉token嵌入的冗余性，减少KV缓存大小，提升解码效率并降低GPU内存消耗。
摘要：
Key-Value (KV) cache已成为现代大型视觉-语言模型（LVLMs）推理中的事实组件。尽管它在大型语言模型（LLMs）中增强了解码效率，但直接在LVLMs中使用会因预填充阶段处理大量视觉token而引入显著的GPU内存开销。为解决此问题，我们提出LightKV，一种新的方法，通过利用视觉token嵌入之间的冗余性来减少KV缓存大小。受文本提示引导，LightKV采用跨模态信息传递来聚合视觉token中的信息性信息，并在预填充过程中逐步压缩它们。这种提示意识的指导使我们的方法区别于以往仅基于视觉的压缩策略。我们在八个开源LVLMs上评估LightKV，跨越八个公共基准数据集，例如MME和SeedBench。实验结果表明，仅使用原始视觉token的55%，LightKV（a）将视觉token的KV缓存大小减半，（b）将计算量减少高达40%，（c）在保持通用性能的同时显著优于现有基线。
英文摘要：
Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage. To tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings. Guided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill. This prompt-aware guidance distinguishes our method from prior vision-only compression strategies. We evaluate LightKV on eight open-source LVLMs across eight public benchmark datasets, e.g., MME and SeedBench. Experimental results demonstrate that with only 55% of the original vision tokens, LightKV (a) halves the vision-token KV cache size, (b) reduces computation by up to 40%, and (c) preserves general-purpose performance while significantly outperforming existing baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
