Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim.
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究所,东部技术研究所);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Munich Center for Machine Learning, LMU Munich(慕尼黑大学机器学习中心,慕尼黑大学)
Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(人工智能与数字经济广东实验室);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);The Hong Kong Polytechnic University(香港理工大学);Great Bay University(大鹏大学);Shenzhen University(深圳大学);Huawei(华为)
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院);Cognitive AI Lab, Shanghai Huawei Technologies, China(上海华为技术有限公司认知人工智能实验室)
UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究所,东部技术研究所);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Shanghai Jiao Tong University(上海交通大学);The Hong Kong Polytechnic University(香港理工大学);Munich Center for Machine Learning, LMU Munich(慕尼黑机器学习中心,慕尼黑大学)
The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
LiAuto Inc.(LiAuto公司);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与移动学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)