多模式协作剪枝用于视觉-语言模型
Collaborative Multi-Mode Pruning for Vision-Language Models
AI总结：
本文提出CoMP框架，通过联合参数和token剪枝，解决单一模式剪枝导致的性能下降问题，通过协作重要性度量和多模式剪枝策略提升高剪枝率下的模型性能。
摘要：
视觉-语言模型（VLMs）在统一的Transformer架构中迅速发展，但其在资源受限设备上的部署仍面临高计算复杂度的挑战。尽管剪枝已成为压缩VLMs的有效技术，但现有方法主要针对单一模式进行参数或token剪枝，忽视了每个模式内在冗余的全面探索，导致高剪枝率下的显著性能下降。为解决上述限制，我们提出协作多模式剪枝（CoMP），一种专为VLMs设计的框架，通过联合参数和token剪枝。具体而言，我们首先设计了协作重要性度量（CIM），研究耦合参数和token之间的相互干扰。它将token的不同重要性纳入参数重要性分数的计算，同时减轻已剪枝参数对token重要性分数的影响。此外，我们开发了多模式剪枝策略（MPS），将整体剪枝过程分解为一系列剪枝阶段，在每个阶段根据剪枝成本估计不同剪枝模式的优先级，并适应性地转向最优模式。此外，MPS整合了历史成本和随机探索，以实现稳定的剪枝过程并避免局部最优。在各种视觉-语言任务和模型上的广泛实验表明，我们的方法通过与最先进的方法比较，有效提升了高剪枝率下的模型性能。源代码可在https://github.com/Wuzimeng/CoMP.git获取。
英文摘要：
Vision-Language Models (VLMs) have advanced rapidly within the unified Transformer architecture, yet their deployment on resource-constrained devices remains challenging due to high computational complexity. While pruning has emerged as an effective technique for compressing VLMs, existing approaches predominantly focus on a single mode by pruning either parameters or tokens, neglecting fully exploring the inherent redundancy in each mode, which leads to substantial performance degradation at high pruning ratios. To address the above limitations, we propose Collaborative Multi-Mode Pruning (CoMP), a novel framework tailored for VLMs by performing joint parameter and token pruning. Specifically, we first design a Collaborative Importance Metric (CIM) that investigates the mutual interference between the coupled parameters and tokens. It incorporates distinct significance of tokens into the computation of parameter importance scores, while simultaneously mitigating the affect of pruned parameters on token importance scores. Moreover, we develop a Multi-Mode Pruning Strategy (MPS) that decomposes the overall pruning process into a sequence of pruning stages, while in each stage we estimate the priory of different pruning modes based on their pruning cost and adaptively shift to the optimal one. Additionally, MPS integrates the historical cost and random exploration, in order to achieve a stable pruning process and avoid local optimum. Extensive experiments across various vision-language tasks and models demonstrate that our method effectively promotes the performance under high pruning ratios by comparing to the state-of-the-art approaches. The source code is available at https://github.com/Wuzimeng/CoMP.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
