TGV-KV：面向视觉语言模型的文本引导KV驱逐方法
TGV-KV: Text-Grounded KV Eviction for Vision-Language Models
AI总结：
针对视觉语言模型中视觉信息冗余导致的KV缓存内存消耗问题，提出基于文本引导的KV驱逐方法TGV-KV，通过文本-视觉预算分配、文本加权排序和文本优先保留策略，在保持高精度的同时显著提升推理吞吐量。
摘要：
视觉语言模型（VLM）继承了自回归生成范式，并缓存所有先前token的键和值（KV）以加速推理，导致内存消耗随上下文长度线性增长。由于视觉模态中存在大量冗余，这一问题在VLM中尤为突出。尽管KV缓存驱逐方法可以有效减少推理内存，但它们通常会导致VLM性能显著下降，因为大多数方法是为语言模型设计的，忽视了文本与视觉之间的固有差距。通过系统分析VLM中的模态差距，我们认为视觉信息的重要性应以文本引导为基础，并据此提出了一种面向VLM的文本引导KV驱逐方法（TGV-KV）。TGV-KV包含三个子模块：（1）文本-视觉预算分配（TVB）基于互信息交互为每层分配预算。（2）文本加权排序（TWR）评估文本的优先级，并根据加权文本-图像注意力对视觉重要性进行排序。（3）文本优先保留（TPR）策略有选择地保留文本KV以避免严重的信息损失。我们在五种不同规模和架构的模型上评估了TGV-KV，结果显示，在LLaVA-NeXT的VizWiz-VQA任务中，TGV-KV保留了全KV准确率的99.2%，并在极端保留预算5%下将端到端吞吐量提升了52.6%。代码可在该https URL获取。
英文摘要：
Vision-Language Models (VLMs) inherit the auto-regressive generation paradigm and cache the keys and values (KV) of all previous tokens to accelerate inference, resulting in memory consumption that scales linearly with context length. This issue is particularly pronounced in VLMs due to substantial redundancy in the visual modality. Although KV cache eviction approaches can effectively reduce inference memory, they often incur significant performance degradation in VLMs, as most are designed for language models and overlook the inherent gap between text and vision. By systematically analyzing the modality gap in VLMs in this work, we argue that the importance of visual information should be grounded in textual guidance and accordingly propose a Text-Grounded KV Eviction method for VLMs (TGV-KV). TGV-KV comprises three submodules: (1) Text-Vision Budgeting (TVB) assigns budget to each layer based on the mutual information interaction. (2) Text-Weighted Ranking (TWR) assesses the priority of text and ranks vision importance based on weighted text-image attention. (3) Text-Prioritised Retention (TPR) policy strategically preserves text KV to avoid acute information loss. We evaluate TGV-KV across five models with different sizes and architectures, showing that TGV-KV preserves 99.2% full-KV accuracy on the VizWiz-VQA task with LLaVA-NeXT and boosts end-to-end throughput by 52.6% with an extreme retention budget of 5%. Code is available at https://github.com/Danielement321/TGV-KV.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
