While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer modules. Sparse adaptive image Transformer (SaiT) offers varying levels of model acceleration by merely changing the token sparsity on the fly. Specifically, SaiT reduces the computation complexity (FLOPs) by 39% - 43% and increases the throughput by 67% - 91% with less than 0.5% accuracy loss for various vision transformer models. Meanwhile, the same model also provides the zero accuracy drop option by skipping the sparsification step. SaiT achieves better accuracy and computation tradeoffs than state-of-the-art transformer and convolutional models.
Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Data Intelligence Institute of Paris（巴黎数据智能研究院）；Université Paris Cité（巴黎西岱大学）；Shijiazhuang Tiedao University（石家庄铁道大学）
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
College of Artificial Intelligence and Green Energy National Yang Ming Chiao Tung University（人工智能与绿色能源学院，阳明交通大学）；Computer Science Department University at Albany, SUNY（计算机科学系，阿尔巴尼大学）；Computer Science Information Engineering Department University at Albany, SUNY（计算机科学与信息工程系，阿尔巴尼大学）
Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
Tongji University（同济大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；East China Normal University（华东师范大学）；Shanghai Eye Disease Prevention and Treatment Center（上海市眼病防治中心）
OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport
Morgan Stanley（摩根士丹利）；Clemson University（克莱姆森大学）；Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Texas A&M University（德克萨斯农工大学）；Rice University（莱斯大学）；Florida State University（佛罗里达州立大学）
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology（浙江省数字化服务技术重点实验室）；The University of Queensland（昆士兰大学）；Singapore Management University（新加坡管理大学）；The University of Southern Queensland（南昆士兰大学）
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models
University of North Carolina Chapel Hill（北卡罗来纳大学教堂山分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Hong Kong University of Science and Technology（香港科学与技术大学）
SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
University of Electronic Science and Technology of China（电子科技大学）；Shenzhen Institute for Advanced Study（深圳先进研究 institute）；CFAR, Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR）认知与人工智能研究中心）；IHPC, Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR）人工智能中心）
The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
LiAuto Inc.（LiAuto公司）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
