Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across a range of multimodal tasks. However, their inference efficiency is constrained by the large number of visual tokens processed during decoding. To address this challenge, we propose Per-Layer Per-Head Vision Token Pruning (PLPHP), a two-level fine-grained pruning method including Layer-Level Retention Rate Allocation and Head-Level Vision Token Pruning. Motivated by the Vision Token Re-attention phenomenon across decoder layers, we dynamically adjust token retention rates layer by layer. Layers that exhibit stronger attention to visual information preserve more vision tokens, while layers with lower vision attention are aggressively pruned. Furthermore, PLPHP applies pruning at the attention head level, enabling different heads within the same layer to independently retain critical context. Experiments on multiple benchmarks demonstrate that PLPHP delivers an 18% faster decoding speed and reduces the Key-Value Cache (KV Cache) size by over 50%, all at the cost of 0.46% average performance drop, while also achieving notable performance improvements in multi-image tasks. These results highlight the effectiveness of fine-grained token pruning and contribute to advancing the efficiency and scalability of LVLMs. Our source code will be made publicly available.
E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models
Digital Environment Research Institute (DERI), Queen Mary University of London（伦敦大学玛丽女王学院数字环境研究所（DERI））；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；National Institute of Education, Nanyang Technological University（南洋理工大学国家教育学院）；School of Computer Science and Engineering, Jiangsu University of Science and Technology（江苏科技大学计算机科学与工程学院）
Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models
Zhejiang Key Laboratory of Space Information Sensing and Transmission, School of Computer Science, Hangzhou Dianzi University（浙江省空间信息感知与传输重点实验室，计算机学院，杭州电子科技大学）；Li Auto Inc.（理想汽车）；School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）
Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning…
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China（东北大学计算机科学与工程学院，沈阳 110819，中国）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China（东北大学计算机与通信工程学院，秦皇岛 066004，中国）
Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究学院）
As language models have grown in parameters and layers, it has become much harder to train and infer with them on single GPUs. This is severely restricting the availability of large language models such as GPT-3, BERT-Large, and many others. A common technique to solve this problem is pruning the network architecture by removing transformer heads, fully-connected weights, and other modules. The main challenge is to discern the important parameters from the less important ones. Our goal is to find strong metrics for identifying such parameters. We thus propose two strategies: Cam-Cut based on t…
Multi-Granular Node Pruning for Causal Circuit Discovery
Department of Computer Science, University of Kentucky, USA（美国肯塔基大学计算机科学系）；Department of Computer Science, Dalhousie University, Canada（加拿大达尔豪斯大学计算机科学系）
CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Interdisciplinary Graduate Programme, Nanyang Technological University（南洋理工大学跨学科研究生项目）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Xiaomi AI Lab（小米AI实验室）
ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
School of Future Technology, Dalian University of Technology（大连理工大学未来技术学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；OPPO Research Institute（OPPO研究院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
Vision transformer (ViT) has achieved competitive accuracy on a variety of computer vision applications, but its computational cost impedes the deployment on resource-limited mobile devices. We explore the sparsity in ViT and observe that informative patches and heads are sufficient for accurate image recognition. In this paper, we propose a cascade pruning framework named CP-ViT by predicting sparsity in ViT models progressively and dynamically to reduce computational redundancy while minimizing the accuracy loss. Specifically, we define the cumulative score to reserve the informative patches…
This paper proposes anchor pruning for object detection in one-stage anchor-based detectors. While pruning techniques are widely used to reduce the computational cost of convolutional neural networks, they tend to focus on optimizing the backbone networks where often most computations are. In this work we demonstrate an additional pruning technique, specifically for object detection: anchor pruning. With more efficient backbone networks and a growing trend of deploying object detectors on embedded systems where post-processing steps such as non-maximum suppression can be a bottleneck, the impa…
Sri Lanka Institute of Information Technology（斯里兰卡信息技术学院）；Ulster University（阿尔斯特大学）
Magnitude-based pruning is a technique used to optimise deep learning models for edge inference. We have achieved over 75% model size reduction with a higher accuracy than the original multi-output regression model for head-pose estimation.
The recently proposed Vision transformers (ViTs) have shown very impressive empirical performance in various computer vision tasks, and they are viewed as an important type of foundation model. However, ViTs are typically constructed with large-scale sizes, which then severely hinder their potential deployment in many practical resources-constrained applications. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enable practical efficiency. However, unlike its current popularity for CNNs and RNNs, structured pruning for ViT models is li…
Graph convolution networks (GCNs) are currently mainstream in learning with irregular data. These models rely on message passing and attention mechanisms that capture context and node-to-node relationships. With multi-head attention, GCNs become highly accurate but oversized, and their deployment on cheap devices requires their pruning. However, pruning at high regimes usually leads to topologically inconsistent networks with weak generalization. In this paper, we devise a novel method for lightweight GCN design. Our proposed approach parses and selects subnetworks with the highest magnitudes …
HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers
School of Natural and Computing Sciences, University of Aberdeen（阿伯丁大学自然与计算科学学院）；School of Computing Sciences, University of East Anglia（东安格利亚大学计算科学学院）；Department of Physics and Technology, UiT The Arctic University of Norway（挪威北极大学物理与技术系）
Learning Diverse Features in Vision Transformers for Improved Generalization
Idiap Research Institute（Idiap研究所）；Mila（米拉人工智能研究所）；University of Montreal（蒙特利尔大学）；University of Bucharest（布加勒斯特大学）；Gheorghe Mihoc - Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy（罗马尼亚科学院Gheorghe Mihoc-Caius Iacob数理统计与应用数学研究所）
