ETC: 通过任务感知的视觉信息蒸馏实现视觉语言模型中的极端令牌压缩
ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMs
AI总结：
提出ETC框架，基于变分信息蒸馏原理，在减少输入令牌数量时最小化任务损失，通过文本-图像交叉注意力加权视觉特征并引入变分信息蒸馏，实现单令牌压缩下仍保持强任务性能。
摘要：
在视觉语言模型（VLM）中，高分辨率图像会产生大量视觉令牌，导致推理时的高计算成本和KV缓存开销。为解决此问题，我们提出极端令牌压缩（ETC）框架，基于变分信息蒸馏原理，在减少输入令牌数量时最小化任务损失。具体而言，从信息论角度，我们表明最小化任务损失需要紧凑表示保留用于预测的指令感知充分统计量。在实践中，ETC利用文本-图像交叉注意力加权原始视觉特征以近似潜在的指令感知预测统计量。此外，ETC引入变分信息蒸馏，使紧凑表示保留必要信息以恢复该预测统计量。在LLaVA-1.5-7B和Qwen3-VL-2B上的实验表明，即使在单令牌压缩下，ETC仍保持有效性，大幅减少KV缓存开销同时保留强任务性能。
英文摘要：
In Vision-Language Models (VLMs), high-resolution images produce a large number of visual tokens, resulting in high computational costs and KV-cache overhead during inference. To address this problem, we propose an Extreme Token Compression (ETC) framework that minimizes task loss when reducing the number of input tokens based on the principle of variational information distillation. Specifically, from an information-theoretic perspective, we show that minimizing task loss requires the compact representation to preserve the instruction-aware sufficient statistic of the task-relevant visual information for prediction. In practice, ETC leverages text-to-image cross-attention to weight the original visual features to approximate the latent instruction-aware predictive statistic. Moreover, ETC introduces a variational information distillation, enabling the compact representation to preserve the essential information to recover this predictive statistic. Experiments on LLaVA-1.5-7B and Qwen3-VL-2B show that ETC remains effective even under single-token compression, substantially reducing KV-cache overhead while retaining strong task performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
