arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13652cs.CL
收藏

LVPruning:一种有效且简单的面向多模态大语言模型的语言引导视觉令牌剪枝方法

LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models

University of Manchester(曼彻斯特大学);Imperial College London(伦敦帝国学院)

Yizheng Sun, Yanze Xin, Hao Li, Jingyuan Sun, Chenghua Lin, Riza Batista-Navarro

AI总结:

本文提出 LVPruning,一种基于语言引导的视觉令牌剪枝方法,通过交叉注意力评估视觉令牌重要性,无需修改模型参数即可显著降低多模态大语言模型的计算开销并保持性能。

摘要:

多模态大语言模型(MLLMs)通过融合视觉与文本模态取得了显著成功。然而,由于需要处理大量视觉令牌,它们带来了可观的计算开销,限制了其在资源受限环境中的实用性。我们提出了面向多模态大语言模型的语言引导视觉令牌剪枝方法(LVPruning),这是一种有效且简单的方法,能够在保持模型性能的同时显著降低计算负担。LVPruning 利用交叉注意力模块,根据视觉令牌与语言令牌的交互来计算视觉令牌的重要性,从而决定剪枝哪些令牌。重要的是,LVPruning 无需修改原始多模态大语言模型参数即可集成,这使得 LVPruning 易于应用或移除。我们的实验表明,LVPruning 在 LLaVA-1.5 的中间层能够有效减少高达 90% 的视觉令牌,使推理每秒万亿次浮点运算(TFLOPs)降低 62.1%,在九个多模态基准上的平均性能损失仅为 0.45%。

英文摘要:

Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑