重新思考视觉语言模型的剪枝：实现有效稀疏性与性能恢复的策略
Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration
AI总结：
针对视觉语言模型部署受限问题，提出SparseLoRA方法将稀疏性直接应用于LoRA权重以恢复剪枝模型性能，在2:4和70%稀疏度下分别实现11.3%和47.6%的提升。
摘要：
视觉语言模型（VLMs）整合来自多种模态的信息，并在各种任务中取得了显著成功。然而，在资源受限的场景中部署大规模 VLMs 具有挑战性。剪枝后接微调提供了一种潜在的解决方案，但在 VLMs 中仍研究不足。本研究解决了两个关键问题：如何在特定于不同模态的模型之间分配稀疏性，以及如何恢复剪枝后稀疏 VLMs 的性能。我们的初步研究确定了两种有效的剪枝设置：对视觉和语言模型应用相同的稀疏度，以及仅剪枝语言模型。虽然 LoRA 微调旨在恢复稀疏模型，但由于与稀疏模型不兼容，它面临挑战，会破坏已剪枝的稀疏性。为了克服这些问题，我们提出了 SparseLoRA，它将稀疏性直接应用于 LoRA 权重。我们的实验结果证明了显著的改进，包括在 2:4 稀疏度下 11.3% 的提升，以及在非结构化 70% 稀疏度下 47.6% 的增强。代码发布于：https://github.com/Shwai-He/VLM-Compression。
英文摘要：
Vision-Language Models (VLMs) integrate information from multiple modalities and have shown remarkable success across various tasks. However, deploying large-scale VLMs in resource-constrained scenarios is challenging. Pruning followed by finetuning offers a potential solution but remains underexplored for VLMs. This study addresses two key questions: how to distribute sparsity across different modality-specific models, and how to restore the performance of pruned sparse VLMs. Our preliminary studies identified two effective pruning settings: applying the same sparsity to both vision and language models, and pruning only the language models. While LoRA finetuning aims to restore sparse models, it faces challenges due to incompatibility with sparse models, disrupting the pruned sparsity. To overcome these issues, we propose SparseLoRA, which applies sparsity directly to LoRA weights. Our experimental results demonstrate significant improvements, including an 11.3\% boost under 2:4 sparsity and a 47.6\% enhancement under unstructured 70\% sparsity. Code is released at: \url{https://github.com/Shwai-He/VLM-Compression}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
