并非所有注意力头都有助于关键视觉标记选择：头感知剪枝更重要

中文摘要 AI 辅助

视觉-语言模型（VLMs）在各类视觉场景中展现出优异性能，但这一成功伴随视觉标记的爆炸式增长，推理阶段会带来巨大的内存和计算开销，最终增加延迟。为提升VLM推理效率，一类典型的视觉标记剪枝方法通过聚合大语言模型（LLM）主干剪枝层中所有注意力头的注意力分数来估计标记重要性，并基于聚合分数剪枝标记。然而，本文揭示了一个引人注目的现象：定位关键视觉标记的能力集中在一小部分注意力头中，仅聚合这些头的注意力分数即可提升任务性能。受此观察启发，我们提出ProViP，一种无需训练的渐进式视觉标记剪枝框架。ProViP首先在LLM主干推理前，基于输入标记的嵌入相似性移除冗余视觉标记，随后在推理过程中通过头感知剪枝进一步剪枝标记。实验表明，ProViP兼具优异的任务性能与推理效率：当应用于LLaVA-1.5-7B时，在88.9%的剪枝率下，ProViP保留了95.9%的原始性能，实现了1.62倍的推理加速。

英文摘要

Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios. However, this success comes at the cost of explosive growth in visual tokens, which imposes substantial memory and computational overhead during inference, ultimately increasing latency. To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance by aggregating attention scores across all heads in the pruning layer of the Large Language Model (LLM) backbone and prunes tokens based on aggregated scores. However, in this paper, we reveal a compelling phenomenon: the capability to pinpoint critical visual tokens is concentrated within a small fraction of heads. Aggregation exclusively on these heads can improve task performance. Inspired by this observation, we propose ProViP, a training-free progressive visual token pruning framework. ProViP first removes redundant visual tokens based on the embedding similarity of input tokens before reasoning of the LLM backbone, and then further prunes tokens during reasoning via head-aware pruning. Experiments demonstrate that ProViP delivers outstanding task performance and inference efficiency. For instance, when applied to LLaVA-1.5-7B, ProViP retains 95.9% of the original performance and achieves 1.62x inference speedup under an 88.9% pruning ratio.

发表机构

• The Hong Kong University of Science and Technology（香港科技大学）
• Northeastern University（东北大学）

机构由 AI 辅助整理，请以论文原文为准。