平衡令牌剪枝：超越局部优化的视觉语言模型加速
Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
AI总结：
针对视觉语言模型令牌剪枝忽视局部与全局影响的问题，提出平衡令牌剪枝（BTP）方法，通过分阶段剪枝兼顾后续层与局部输出一致性，实现78%压缩率并保留96.7%性能。
摘要：
大型视觉语言模型（LVLMs）通过将图像编码为数千个令牌，在多模态任务中展现出令人瞩目的性能。然而，大量的图像令牌导致显著的计算开销，而动态高分辨率输入的使用进一步加重了这一负担。以往的方法尝试通过令牌剪枝来减少图像令牌数量，通常基于注意力分数或图像令牌多样性来选择令牌。通过实证研究，我们观察到现有方法往往忽视了剪枝对当前层输出（局部）和后续层输出（全局）的联合影响，导致剪枝决策次优。为解决这一挑战，我们提出了平衡令牌剪枝（BTP），一种即插即用的视觉令牌剪枝方法。具体而言，我们的方法利用一个小型校准集将剪枝过程划分为多个阶段。在早期阶段，我们的方法强调剪枝对后续层的影响，而在较深阶段，重点转向保持局部输出的一致性。在多种LVLMs上的大量实验证明了我们方法在多个基准上的广泛有效性。我们的方法在平均保留原始模型96.7%性能的同时，实现了78%的压缩率。我们的代码可在https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning获取。
英文摘要：
Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens. However, the large number of image tokens results in significant computational overhead, and the use of dynamic high-resolution inputs further increases this burden. Previous approaches have attempted to reduce the number of image tokens through token pruning, typically by selecting tokens based on attention scores or image token diversity. Through empirical studies, we observe that existing methods often overlook the joint impact of pruning on both the current layer's output (local) and the outputs of subsequent layers (global), leading to suboptimal pruning decisions. To address this challenge, we propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens. Specifically, our method utilizes a small calibration set to divide the pruning process into multiple stages. In the early stages, our method emphasizes the impact of pruning on subsequent layers, whereas in the deeper stages, the focus shifts toward preserving the consistency of local outputs. Extensive experiments across various LVLMs demonstrate the broad effectiveness of our approach on multiple benchmarks. Our method achieves a 78% compression rate while preserving 96.7% of the original models' performance on average. Our code is available at https://github.com/EmbodiedCity/NeurIPS2025-Balanced-Token-Pruning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
