Vision transformers have achieved leading performance on various visual tasks yet still suffer from high computational complexity. The situation deteriorates in dense prediction tasks like semantic segmentation, as high-resolution inputs and outputs usually imply more tokens involved in computations. Directly removing the less attentive tokens has been discussed for the image classification task but can not be extended to semantic segmentation since a dense prediction is required for every patch. To this end, this work introduces a Dynamic Token Pruning (DToP) method based on the early exit of tokens for semantic segmentation. Motivated by the coarse-to-fine segmentation process by humans, we naturally split the widely adopted auxiliary-loss-based network architecture into several stages, where each auxiliary block grades every token's difficulty level. We can finalize the prediction of easy tokens in advance without completing the entire forward pass. Moreover, we keep $k$ highest confidence tokens for each semantic category to uphold the representative context information. Thus, computational complexity will change with the difficulty of the input, akin to the way humans do segmentation. Experiments suggest that the proposed DToP architecture reduces on average $20\% - 35\%$ of computational cost for current semantic segmentation methods based on plain vision transformers without accuracy degradation.
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京,中国);Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd, Beijing, China(凡语AI实验室,中科创始人技术有限公司,北京,中国)
When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
Tongji University(同济大学);University of California, Santa Cruz(加州大学圣克ruz分校);Amazon(亚马逊);East China Normal University(华东师范大学);Shanghai Eye Disease Prevention and Treatment Center(上海眼病预防与治疗中心)
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Global Innovation Exchange, Tsinghua University(清华大学全球创新交流中心);Department of Automation, Tsinghua University(清华大学自动化系);School of Computer Science, Peking University(北京大学计算机科学系)
ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea(电气与计算机工程系,成均馆大学,大韩民国);Department of Semiconductor Systems Engineering, Sungkyunkwan University, Republic of Korea(半导体系统工程系,成均馆大学,大韩民国)
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
School of Information Engineering, Chang’an University, China(长安大学信息工程学院);School of Artificial Intelligence, Shanghai Jiaotong University, China(上海交通大学人工智能学院)
Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach
School of Computer Science, Fudan University(复旦大学计算机科学学院);Data Intelligence Institute of Paris(巴黎数据智能研究院);Université Paris Cité(巴黎西岱大学);Shijiazhuang Tiedao University(石家庄铁道大学)
Attention Debiasing for Token Pruning in Vision Language Models
School of Communication and Information Engineering, Shanghai University(通信与信息工程学院,上海大学);School of Computer Science and Engineering, Shanghai University(计算机科学与工程学院,上海大学);School of Computer Science and Engineering, Nankai University(计算机科学与工程学院,南开大学)
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
College of Artificial Intelligence and Green Energy National Yang Ming Chiao Tung University(人工智能与绿色能源学院,阳明交通大学);Computer Science Department University at Albany, SUNY(计算机科学系,阿尔巴尼大学);Computer Science Information Engineering Department University at Albany, SUNY(计算机科学与信息工程系,阿尔巴尼大学)
Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
Tongji University(同济大学);University of California, Santa Cruz(加州大学圣克鲁兹分校);East China Normal University(华东师范大学);Shanghai Eye Disease Prevention and Treatment Center(上海市眼病防治中心)
Towards Joint Quantization and Token Pruning of Vision-Language Models
VCIP, College of Computer Science, Nankai University(VCIP,计算机科学学院,南开大学);School of Computer Science and Engineering, Tianjin University of Technology(计算机科学与工程学院,天津工业大学);OPPO Research Institute(OPPO研究院);Academy for Advanced Interdisciplinary Studies, Nankai University(先进跨学科研究院,南开大学);Nankai International Advanced Research Institute, Shenzhen Futian(南开国际先进研究 institutes,深圳福田);Department of Computing, Hong Kong Polytechnic University(计算学院,香港理工大学)
OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport
Morgan Stanley(摩根士丹利);Clemson University(克莱姆森大学);Arizona State University(亚利桑那州立大学);Washington University in St. Louis(圣路易斯华盛顿大学);Texas A&M University(德克萨斯农工大学);Rice University(莱斯大学);Florida State University(佛罗里达州立大学)