arXiv 2508.172432025-08-30cs.CVcs.AIcs.CL

CoViPAL：面向大视觉语言模型的层级上下文化视觉令牌剪枝

CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models

School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）；Cognitive AI Lab, Shanghai Huawei Technologies, China（上海华为技术有限公司认知人工智能实验室）

Zicong Tang, Ziyang Ma, Suqing Wang, Zuchao Li, Lefei Zhang, Hai Zhao, Yun Li, Qianren Wang

备注：: Accepted by EMNLP 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大视觉语言模型视觉令牌过多导致的计算与内存开销问题，提出即插即用的层级上下文化剪枝方法CoViPAL，在浅层即可有效剪枝冗余令牌，在多基准上性能优于同类免训练及同监督水平的训练型剪枝方法，兼顾推理效率与精度。

摘要：

大视觉语言模型（LVLM）处理由文本令牌以及从图像或视频中提取的视觉令牌组成的多模态输入。由于视觉信息丰富，单张图像可生成数千个视觉令牌，导致预填充阶段计算成本高昂，且解码阶段内存开销巨大。现有方法尝试剪枝冗余视觉令牌，证实了视觉表征中存在大量冗余，但这类方法往往因缺乏足够上下文信息，在浅层网络中效果不佳。我们认为，许多视觉令牌即便在浅层也具有固有冗余性，借助恰当的上下文信号即可安全有效地完成剪枝。本文提出CoViPAL——一种层级上下文化视觉令牌剪枝方法，它采用即插即用剪枝模块（Plug-and-Play Pruning Module，PPM），在冗余视觉令牌被LVLM处理之前对其进行预测和移除。PPM轻量、模型无关，且独立于LVLM架构运行，可无缝集成到多种模型中。在多个基准上开展的大量实验表明，在相同令牌预算下，CoViPAL的性能优于免训练剪枝方法；在监督程度相当的情况下，其表现也超过了基于训练的剪枝方法。CoViPAL为提升LVLM的推理效率提供了一种可扩展且高效的方案，同时不会损失精度。

英文摘要：

Large Vision-Language Models (LVLMs) process multimodal inputs consisting of text tokens and vision tokens extracted from images or videos. Due to the rich visual information, a single image can generate thousands of vision tokens, leading to high computational costs during the prefilling stage and significant memory overhead during decoding. Existing methods attempt to prune redundant vision tokens, revealing substantial redundancy in visual representations. However, these methods often struggle in shallow layers due to the lack of sufficient contextual information. We argue that many visual tokens are inherently redundant even in shallow layers and can be safely and effectively pruned with appropriate contextual signals. In this work, we propose CoViPAL, a layer-wise contextualized visual token pruning method that employs a Plug-and-Play Pruning Module (PPM) to predict and remove redundant vision tokens before they are processed by the LVLM. The PPM is lightweight, model-agnostic, and operates independently of the LVLM architecture, ensuring seamless integration with various models. Extensive experiments on multiple benchmarks demonstrate that CoViPAL outperforms training-free pruning methods under equal token budgets and surpasses training-based methods with comparable supervision. CoViPAL offers a scalable and efficient solution to improve inference efficiency in LVLMs without compromising accuracy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。