Multi-modal Large Language Models (MLLMs) have achieved remarkable success by integrating visual and textual modalities. However, they incur significant computational overhead due to the large number of vision tokens processed, limiting their practicality in resource-constrained environments. We introduce Language-Guided Vision Token Pruning (LVPruning) for MLLMs, an effective yet simple method that significantly reduces the computational burden while preserving model performance. LVPruning employs cross-attention modules to compute the importance of vision tokens based on their interaction with language tokens, determining which to prune. Importantly, LVPruning can be integrated without modifying the original MLLM parameters, which makes LVPruning simple to apply or remove. Our experiments show that LVPruning can effectively reduce up to 90% of vision tokens by the middle layer of LLaVA-1.5, resulting in a 62.1% decrease in inference Tera Floating-Point Operations Per Second (TFLOPs), with an average performance loss of just 0.45% across nine multi-modal benchmarks.
$\mathcal{V}isi\mathcal{P}runer$: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT, Ningbo(宁波空间智能与数字衍生关键实验室,数字孪生研究院,EIT,宁波);Shanghai Jiao Tong University(上海交通大学);Hong Kong Polytechnic University(香港理工大学);Meituan Inc.(美团公司);National University of Singapore(新加坡国立大学)
OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)(模式识别新实验室(NLPR)、自动化研究所、中国科学院(CASIA));Nanjing University(南京大学);The Hong Kong University of Science(香港科学大学);Sichuan University(四川大学);Peking University(北京大学)
G-Refer: Graph Retrieval-Augmented Large Language Model for Explainable Recommendation
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Chinese University of Hong Kong(香港中文大学);Monash University(莫纳什大学);Huawei Technologies Co., Ltd.(华为技术有限公司)
Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring
School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Harbin Institute of Technology, Shenzhen(深圳哈尔滨工业大学);University of Science and Technology Beijing(北京科技大学)
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
EPIC Lab, Shanghai Jiao Tong University(上海交通大学EPIC实验室);DAIL Tech(DAIL科技);Northeastern University(东北大学);Sichuan University(四川大学);Huazhong University of Science and Technology(华中科技大学)