VLM-Pruner: 为高效VLM离心令牌剪枝范式中的空间稀疏性引入缓冲
VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm
AI总结：
VLM-Pruner通过平衡冗余与空间稀疏性，提升VLMs的效率和性能。
摘要：
视觉-语言模型（VLMs）在图像理解任务中表现出色，但大量的视觉令牌数量带来了显著的计算成本，阻碍了在移动设备上的部署。许多剪枝方法仅依赖于令牌重要性，因此忽略了令牌间的冗余，保留了许多重复的令牌并浪费了容量。虽然已经提出了一些冗余意识的方法，但它们通常忽略了视觉令牌之间的空间关系。这可能导致对保留令牌的选择过于稀疏，无法充分覆盖目标对象的区域。为了解决这些限制，我们提出了VLM-Pruner，一种无需训练的令牌剪枝算法，它明确地平衡了冗余和空间稀疏性。我们引入了一种离心令牌剪枝范式，使近到远的选择能够在优先保留细粒度对象细节的同时进行。此外，我们设计了一个用于空间稀疏性的缓冲（BSS）准则，该准则延迟了空间上遥远令牌的选择。我们进一步采用并行贪心策略来高效地进行令牌选择。为了减轻剪枝带来的信息损失，我们选择性地将丢弃令牌中的显著信息融合到保留的令牌中。全面的比较表明，VLM-Pruner在五个VLMs上以88.9%的剪枝率 consistently 超过了强大的基线，同时实现了端到端的推理加速。代码可在https://github.com/Casey-bit/VLMPruner上获得。
英文摘要：
Vision-language models (VLMs) excel at image understanding tasks, but the large number of visual tokens imposes significant computational costs, hindering deployment on mobile devices. Many pruning methods rely solely on token importance and thus overlook inter-token redundancy, retaining numerous duplicated tokens and wasting capacity. Although some redundancy-aware approaches have been proposed, they often ignore the spatial relationships among visual tokens. This can lead to overly sparse selections of retained tokens that fail to adequately cover the regions of target objects. To address these limitations, we propose VLM-Pruner, a training-free token pruning algorithm that explicitly balances redundancy and spatial sparsity. We introduce a centrifugal token pruning paradigm that enables near-to-far selection while prioritizing the preservation of fine-grained object details. Moreover, we design a Buffering for Spatial Sparsity (BSS) criterion that defers the selection of spatially distant tokens. We further adopt a parallel greedy strategy to conduct token selection efficiently. To mitigate information loss from pruning, we selectively fuse salient information from the discarded tokens into the retained ones. Comprehensive comparisons demonstrate that VLM-Pruner consistently outperforms strong baselines across five VLMs with an 88.9\% pruning rate, while delivering an end-to-end inference speedup. The code is available at https://github.com/Casey-bit/VLMPruner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
