TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers
State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；College of Computer Science and Technology（计算机科学与技术学院）；College of Computer Science and Electronic Engineering（计算机科学与电子工程学院）
Vision Transformers (ViTs) have demonstrated strong performance across a wide range of vision tasks, yet their substantial computational and memory demands hinder efficient deployment on resource-constrained mobile and edge devices. Pruning has emerged as a promising direction for reducing ViT complexity. However, existing approaches either (i) produce a single pruned model shared across all devices, ignoring device heterogeneity, or (ii) rely on fine-tuning with device-local data, which is often infeasible due to limited on-device resources and strict privacy constraints. As a result, current methods fall short of enabling task-customized ViT pruning in privacy-preserving mobile computing settings. This paper introduces TAP-ViTs, a novel task-adaptive pruning framework that generates device-specific pruned ViT models without requiring access to any raw local data. Specifically, to infer device-level task characteristics under privacy constraints, we propose a Gaussian Mixture Model (GMM)-based metric dataset construction mechanism. Each device fits a lightweight GMM to approximate its private data distribution and uploads only the GMM parameters. Using these parameters, the cloud selects distribution-consistent samples from public data to construct a task-representative metric dataset for each device. Based on this proxy dataset, we further develop a dual-granularity importance evaluation-based pruning strategy that jointly measures composite neuron importance and adaptive layer importance, enabling fine-grained, task-aware pruning tailored to each device's computational budget. Extensive experiments across multiple ViT backbones and datasets demonstrate that TAP-ViTs consistently outperforms state-of-the-art pruning methods under comparable compression ratios.
Neural network pruning is an important step in design process of efficient neural networks for edge devices with limited computational power. Pruning is a form of knowledge transfer from the weights of the original network to a smaller target subnetwork. We propose a new method for compute-constrained structured channel-wise pruning of convolutional neural networks. The method iteratively fine-tunes the network, while gradually tapering the computation resources available to the pruned network via a holonomic constraint in the method of Lagrangian multipliers framework. An explicit and adaptiv…
While vision transformers (ViTs) have continuously achieved new milestones in the field of computer vision, their sophisticated network architectures with high computation and memory costs have impeded their deployment on resource-limited edge devices. In this paper, we propose a hardware-efficient image-adaptive token pruning framework called HeatViT for efficient yet accurate ViT acceleration on embedded FPGAs. By analyzing the inherent computational patterns in ViTs, we first design an effective attention-based multi-head token selector, which can be progressively inserted before transforme…
The recently proposed Vision transformers (ViTs) have shown very impressive empirical performance in various computer vision tasks, and they are viewed as an important type of foundation model. However, ViTs are typically constructed with large-scale sizes, which then severely hinder their potential deployment in many practical resources-constrained applications. To mitigate this challenging problem, structured pruning is a promising solution to compress model size and enable practical efficiency. However, unlike its current popularity for CNNs and RNNs, structured pruning for ViT models is li…
Northeastern University（东北大学）；Clemson University（克莱姆森大学）；Peking University（北京大学）；College of William and Mary（威廉与玛丽学院）；ETH Zürich（苏黎世联邦理工学院）
Recently, Vision Transformer (ViT) has continuously established new milestones in the computer vision field, while the high computation and memory cost makes its propagation in industrial production difficult. Pruning, a traditional model compression paradigm for hardware efficiency, has been widely applied in various DNN structures. Nevertheless, it stays ambiguous on how to perform exclusive pruning on the ViT structure. Considering three key points: the structural characteristics, the internal data pattern of ViTs, and the related edge device deployment, we leverage the input token sparsity…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Texas A&M University（德州农工大学）；Kwai Inc.（快手公司）
Vision transformers (ViTs) have gained popularity recently. Even without customized image operators such as convolutions, ViTs can yield competitive performance when properly trained on massive data. However, the computational overhead of ViTs remains prohibitive, due to stacking multi-head self-attention modules and else. Compared to the vast literature and prevailing success in compressing convolutional neural networks, the study of Vision Transformer compression has also just emerged, and existing works focused on one or two aspects of compression. This paper proposes a unified ViT compress…
DORA: Dynamic Online Reinforcement Agent for Token Pruning in Vision Transformers
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Recently, vision transformer (ViT) and its variants have achieved promising performances in various computer vision tasks. Yet the high computational costs and training data requirements of ViTs limit their application in resource-constrained settings. Model compression is an effective method to speed up deep learning models, but the research of compressing ViTs has been less explored. Many previous works concentrate on reducing the number of tokens. However, this line of attack breaks down the spatial structure of ViTs and is hard to be generalized into downstream tasks. In this paper, we des…
Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning
Xidian University（西安电子科技大学）；University of Michigan – Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；Shanghai Jiao Tong University（上海交通大学）；Norwegian University of Science and Technology（挪威科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）
Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Data Intelligence Institute of Paris（巴黎数据智能研究院）；Université Paris Cité（巴黎西岱大学）；Shijiazhuang Tiedao University（石家庄铁道大学）
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
College of Artificial Intelligence and Green Energy National Yang Ming Chiao Tung University（人工智能与绿色能源学院，阳明交通大学）；Computer Science Department University at Albany, SUNY（计算机科学系，阿尔巴尼大学）；Computer Science Information Engineering Department University at Albany, SUNY（计算机科学与信息工程系，阿尔巴尼大学）
NVIDIA（英伟达）；University of California, Berkeley（加州大学伯克利分校）；Duke University（杜克大学）
Transformers yield state-of-the-art results across many tasks. However, their heuristically designed architecture impose huge computational costs during inference. This work aims on challenging the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage, where we redistribute the parameters both across transformer blocks and between different structures within the block via the first systematic attempt on global structural pruning. Dealing with diverse ViT structural components, we derive a novel Hessian-based structu…
NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devices
National Engineering Research Center for Big Data Technology and System, Services Computing Technology and System Lab, Cluster and Grid Computing Lab（大数据技术与系统国家工程研究中心、服务计算技术与系统实验室、集群与网格计算实验室）；Swinburne University of Technology（斯威本科技大学）；Deakin University（迪金大学）
Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks
School of Mathematics, Southeast University（东南大学数学学院）；Systems Research Institute of the Polish Academy of Sciences（波兰科学院系统研究所）；School of Cyber Science & Engineering, Southeast University（东南大学网络科学与工程学院）；Institute of Computer Science, AGH University of Krakow（克拉科夫AGH大学计算机科学学院）；SAN University（SAN大学）
Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院哈索·普拉特纳数字健康研究所）；Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；Google DeepMind（谷歌DeepMind）
Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This ne…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Microsoft Corporation（微软公司）；International Digital Economy Academy（国际数字经济学院）
Vision transformers (ViTs) have recently received explosive popularity, but their enormous model sizes and training costs remain daunting. Conventional post-training pruning often incurs higher training budgets. In contrast, this paper aims to trim down both the training memory overhead and the inference complexity, without sacrificing the achievable accuracy. We carry out the first-of-its-kind comprehensive exploration, on taking a unified approach of integrating sparsity in ViTs "from end to end". Specifically, instead of training full ViTs, we dynamically extract and train sparse subnetwork…
University of California, Davis（加州大学戴维斯分校）；Visa Research（Visa研究院）
Vision transformer (ViT) expands the success of transformer models from sequential data to images. The model decomposes an image into many smaller patches and arranges them into a sequence. Multi-head self-attentions are then applied to the sequence to learn the attention between patches. Despite many successful interpretations of transformers on sequential data, little effort has been devoted to the interpretation of ViTs, and many questions remain unanswered. For example, among the numerous attention heads, which one is more important? How strong are individual patches attending to their spa…
Learning Diverse Features in Vision Transformers for Improved Generalization
Idiap Research Institute（Idiap研究所）；Mila（米拉人工智能研究所）；University of Montreal（蒙特利尔大学）；University of Bucharest（布加勒斯特大学）；Gheorghe Mihoc - Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy（罗马尼亚科学院Gheorghe Mihoc-Caius Iacob数理统计与应用数学研究所）
From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition
Aalborg University（奥尔堡大学）；Aarhus University（奥胡斯大学）；Poineer Centre for AI（先锋人工智能中心）；University of Southern Denmark（南丹麦大学）；Milestone System（Milestone系统公司）
