ViTCoP: 通过视觉和文本语义协同剪枝加速大型视觉-语言模型
ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning
AI总结：
ViTCoP通过结合视觉和文本语义协同剪枝，有效降低大型视觉-语言模型的计算成本，提升推理效率和内存利用率。
摘要：
大型视觉-语言模型（LVLMs）由于其视觉token中存在显著的冗余性而带来了高昂的计算成本。为了有效降低这一成本，研究者提出了各种视觉token剪枝方法。然而，现有方法通常存在局限，要么由于在视觉编码器中的剪枝而过早丢失关键的视觉信息，要么由于在大型语言模型（LLMs）中的剪枝而导致所选token之间出现信息冗余。为了解决这些挑战，我们提出了一种视觉和文本语义协同剪枝框架（ViTCoP），该框架结合了视觉编码器中的冗余过滤与基于其分层特性的逐步协同剪枝，以高效地保留关键且信息多样化的视觉token。同时，为了确保与如FlashAttention等加速技术的兼容性，我们引入了LLM中的K向量L2范数作为token显著性度量。在各种大型视觉-语言模型上的广泛实验表明，ViTCoP不仅在图像和视频理解任务上超越了现有方法，还显著减少了模型推理延迟和GPU内存消耗。值得注意的是，其在极端剪枝率下的性能优势尤为突出。
英文摘要：
Large Vision-Language Models (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the Large Language Models (LLMs). To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-Language Models demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
