LearnPruner: 重新思考基于注意力的视觉语言模型中令牌修剪
LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
AI总结：
本文提出LearnPruner，通过两阶段令牌修剪框架在视觉语言模型中实现高效修剪，保留95%性能的同时减少5.5%视觉令牌使用，提升3.2倍推理速度。
摘要：
视觉语言模型（VLMs）在视觉理解和推理方面表现出色，但长视觉序列输入带来显著计算负担。近期研究通过修剪不重要视觉令牌实现计算减少。本文分析了视觉编码器和大语言模型（LLM）中注意力机制的有效性，发现视觉编码器存在注意力沉底问题，导致对信息性前景区域关注不足；而LLM中虽然存在注意力偏向于令牌位置的现象，但文本到视觉注意力对这种偏见具有抵抗性，并在中间层实现有效的修剪指导。基于此，我们提出LearnPruner，一个两阶段令牌修剪框架，首先通过可学习的修剪模块在视觉编码器后去除冗余视觉令牌，然后在LLM中间层保留任务相关的令牌。实验结果表明，LearnPruner可在保留约95%原始性能的同时仅使用5.5%的视觉令牌，并实现3.2倍的推理加速，展示了更优的准确率-效率权衡。
英文摘要：
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning, but they also impose significant computational burdens due to long visual sequence inputs. Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction while maintaining model performance. The core of token pruning lies in determining token importance, with current approaches primarily relying on attention scores from vision encoders or Large Language Models (LLMs). In this paper, we analyze the effectiveness of attention mechanisms in both vision encoders and LLMs. We find that vision encoders suffer from attention sink, leading to poor focus on informative foreground regions, while in LLMs, although prior studies have identified attention bias toward token positions, text-to-vision attention demonstrates resistance to this bias and enables effective pruning guidance in middle layers. Based on these observations, we propose LearnPruner, a two-stage token pruning framework that first removes redundant vision tokens via a learnable pruning module after the vision encoder, then retains only task-relevant tokens in the LLM's middle layer. Experimental results show that our LearnPruner can preserve approximately 95% of the original performance while using only 5.5% of vision tokens, and achieve 3.2$\times$ inference acceleration, demonstrating a superior accuracy-efficiency trade-off.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
