EfficientLLaVA：面向大型视觉-语言模型的可泛化自动剪枝
EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models
AI总结：
针对大型视觉-语言模型部署复杂度高的问题，提出仅需少量样本的自动剪枝方法EfficientLLaVA，通过结构风险最小化优化剪枝策略泛化性，在视觉问答任务上实现精度与效率的更优权衡。
摘要：
尽管多模态大语言模型在复杂推理任务中展现出强劲性能，但它们在部署阶段面临模型复杂度带来的严峻挑战，尤其对于资源受限设备而言问题更为突出。本文提出一种面向大型视觉-语言模型的自动剪枝方法，以提升多模态推理的效率。传统方法依赖原模型的训练数据为不同网络组件选择合适的剪枝比例，但由于网页级训练语料带来的难以承受的搜索成本，这类方法对大型视觉-语言模型并不实用。与之不同，我们的方法仅利用少量样本搜索理想剪枝策略，通过最大化其在未知训练数据上的泛化能力同时保持模型精度，实现大型视觉-语言模型精度与效率的最优权衡。具体而言，我们基于结构风险最小化原理对剪枝策略的泛化间隙进行建模。结合任务性能与泛化能力，我们在给定搜索空间内迭代搜索最优剪枝策略，同时优化视觉投影器以演化出性能上限更高的搜索空间。我们在ScienceQA、Vizwiz、MM-vet和LLaVA-Bench数据集上针对视觉问答任务开展了大量实验。仅使用64个样本进行剪枝策略搜索，EfficientLLaVA在ScienceQA上达到83.05%的准确率，同时相比稠密LLaVA-v1.5-7B模型实现了1.8倍的速度提升。
英文摘要：
While multimodal large language models demonstrate strong performance in complex reasoning tasks, they pose significant challenges related to model complexity during deployment, especially for resource-limited devices. In this paper, we propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning. Conventional methods rely on the training data of the original model to select the proper pruning ratio for different network components. However, these methods are impractical for large vision-language models due to the unaffordable search costs caused by web-scale training corpus. In contrast, our approach only leverages a small number of samples to search for the desired pruning policy by maximizing its generalization ability on unknown training data while maintaining the model accuracy, which enables the achievement of an optimal trade-off between accuracy and efficiency for large visual language models. Specifically, we formulate the generalization gap of the pruning strategy using the structural risk minimization principle. Based on both task performance and generalization capability, we iteratively search for the optimal pruning policy within a given search space and optimize the vision projector to evolve the search space with higher upper bound of performance. We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering. Using only 64 samples for pruning policy search, EfficientLLaVA achieves an accuracy of 83.05% on ScienceQA, along with a $\times$ 1.8 speedup compared to the dense LLaVA-v1.5-7B model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
