不要只追逐 MLLM 中的“高亮标记”：重新审视视觉整体上下文保留
Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
AI总结：
针对MLLM视觉标记剪枝中注意力优先方法导致高剪枝率下性能下降的问题，提出整体视角的即插即用剪枝框架HoloV，通过自适应分配剪枝预算保留全局上下文，在剪除88.9%标记时保持95.8%性能。
摘要：
尽管多模态大语言模型（MLLMs）具有强大的能力，但由于其依赖海量的视觉标记，它们承受着相当大的计算开销。近年来的研究探索了标记剪枝以缓解这一问题，通常使用文本-视觉交叉注意力或 [\ exttt{CLS}] 注意力来评估并丢弃冗余的视觉标记。在这项工作中，我们识别了此类注意力优先剪枝方法的一个关键局限性，即它们倾向于保留语义相似的标记，导致在高剪枝比率下出现显著的性能下降。为此，我们提出了 HoloV，一个简单而有效、即插即用的视觉标记剪枝框架，用于高效推理。与之前的注意力优先方案不同，HoloV 从整体视角重新思考标记保留。通过在不同空间裁剪区域间自适应地分配剪枝预算，HoloV 确保保留的标记捕获全局视觉上下文，而非孤立的显著特征。该策略最小化表示坍缩，并在激进剪枝下维持与任务相关的信息。实验结果表明，与最先进方法相比，我们的 HoloV 在各种任务、MLLM 架构和剪枝比率下均实现了优越的性能。例如，配备 HoloV 的 LLaVA1.5 在剪除 88.9% 的视觉标记后保留了原始性能的 95.8%，实现了优越的效率-准确性权衡。
英文摘要：
Despite their powerful capabilities, Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens. Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention or [\texttt{CLS}] attention to assess and discard redundant visual tokens. In this work, we identify a critical limitation of such attention-first pruning approaches, i.e., they tend to preserve semantically similar tokens, resulting in pronounced performance drops under high pruning ratios. To this end, we propose {HoloV}, a simple yet effective, plug-and-play visual token pruning framework for efficient inference. Distinct from previous attention-first schemes, HoloV rethinks token retention from a holistic perspective. By adaptively distributing the pruning budget across different spatial crops, HoloV ensures that the retained tokens capture the global visual context rather than isolated salient features. This strategy minimizes representational collapse and maintains task-relevant information even under aggressive pruning. Experimental results demonstrate that our HoloV achieves superior performance across various tasks, MLLM architectures, and pruning ratios compared to SOTA methods. For instance, LLaVA1.5 equipped with HoloV preserves 95.8\% of the original performance after pruning 88.9\% of visual tokens, achieving superior efficiency-accuracy trade-offs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
