We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.
The landscape of image generation has been forever changed by open vocabulary diffusion models. However, at their core these models use transformers, which makes generation slow. Better implementations to increase the throughput of these transformers have emerged, but they still evaluate the entire model. In this paper, we instead speed up diffusion models by exploiting natural redundancy in generated images by merging redundant tokens. After making some diffusion-specific improvements to Token Merging (ToMe), our ToMe for Stable Diffusion can reduce the number of tokens in an existing Stable …
Yitu Tech（依图科技）；National University of Singapore（新加坡国立大学）；YITU Technology Institute of Data Science（依图科技数据科学研究所）
Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neig…
LapMerge: Differentiable Exact-Budget Token Merging in Linear Time
Jagiellonian University in Kraków（克拉科夫雅盖隆大学）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Centre for Credible AI, Warsaw University of Technology（华沙理工大学可信人工智能中心）
Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）
MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
Tsinghua University（清华大学）；Westlake University（西湖大学）；Zhejiang University（浙江大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；OPPO；Shanghai AI Lab（上海人工智能实验室）
We introduce a novel paradigm for offline Video Instance Segmentation (VIS), based on the hypothesis that explicit object-oriented information can be a strong clue for understanding the context of the entire sequence. To this end, we propose VITA, a simple structure built on top of an off-the-shelf Transformer-based image instance segmentation model. Specifically, we use an image object detector as a means of distilling object-specific contexts into object tokens. VITA accomplishes video-level understanding by associating frame-level object tokens without using spatio-temporal backbone feature…
Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea（韩国科学技术院金载哲人工智能研究生院，大田，韩国）；School of Computing, KAIST, Daejeon, South Korea（韩国科学技术院计算学院，大田，韩国）
ReToMe-VA: Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）
Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Pengcheng Laboratory（鹏城实验室）；Ant Group（蚂蚁集团）
Midea Group, AI Innovation Center（美的集团人工智能创新中心）；Tongji University（同济大学）
The vision transformer splits each image into a sequence of tokens with fixed length and processes the tokens in the same way as words in natural language processing. More tokens normally lead to better performance but considerably increased computational cost. Motivated by the proverb "A picture is worth a thousand words" we aim to accelerate the ViT model by making a long image short. To this end, we propose a novel approach to assign token length adaptively during inference. Specifically, we first train a ViT model, called Resizable-ViT (ReViT), that can process any given input with diverse…
DORA: Dynamic Online Reinforcement Agent for Token Merging in Vision Transformers
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥国家科学中心人工智能研究院）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Jiao Tong University（上海交通大学）；Tencent Youtu Lab（腾讯优图实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) …
Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National Cheng-Kung University（国立成功大学）
针对 DiT 推理计算昂贵的问题，本文提出训练后即插即用的 SDTM，依据“先结构后细节”先验动态合并冗余视觉 token，实现 1.55 倍加速且几乎不损失图像质量。
Recently, several Vision Transformer (ViT) based methods have been proposed for Fine-Grained Visual Classification (FGVC).These methods significantly surpass existing CNN-based ones, demonstrating the effectiveness of ViT in FGVC tasks.However, there are some limitations when applying ViT directly to FGVC.First, ViT needs to split images into patches and calculate the attention of every pair, which may result in heavy redundant calculation and unsatisfying performance when handling fine-grained images with complex background and small objects.Second, a standard ViT only utilizes the class toke…
LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging
Nanjing University of Posts and Telecommunications（南京邮电大学）；Horizon Robotics；Nanjing University（南京大学）；Zhejiang University（浙江大学）；Macau University of Science and Technology（澳门科技大学）；TARS Robotics；China Mobile Zijin Innovation Institute（中国移动吉林创新研究院）
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
College of Artificial Intelligence and Green Energy National Yang Ming Chiao Tung University（人工智能与绿色能源学院，阳明交通大学）；Computer Science Department University at Albany, SUNY（计算机科学系，阿尔巴尼大学）；Computer Science Information Engineering Department University at Albany, SUNY（计算机科学与信息工程系，阿尔巴尼大学）
