Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by the massive number of tokens across the entire video frames, which has been the major barrier to training the model. Further, the patches irrelevant to the main contents, e.g., backgrounds, degrade the generalization performance of models. To tackle these issues, we propose training free token merging for lightweight video Transformer (vid-TLDR) that aims to enhance the efficiency of video Transformers by merging the background tokens without additional training. For vid-TLDR, we introduce a novel approach to capture the salient regions in videos only with the attention map. Further, we introduce the saliency-aware token merging strategy by dropping the background tokens and sharpening the object scores. Our experiments show that vid-TLDR significantly mitigates the computational complexity of video Transformers while achieving competitive performance compared to the base model without vid-TLDR. Code is available at https://github.com/mlvlab/vid-TLDR.
We address semi-supervised video object segmentation, the task of automatically generating accurate and consistent pixel masks for objects in a video sequence, given the first-frame ground truth annotations. Towards this goal, we present the PReMVOS algorithm (Proposal-generation, Refinement and Merging for Video Object Segmentation). Our method separates this problem into two steps, first generating a set of accurate object segmentation mask proposals for each video frame and then selecting and merging these proposals into accurate and temporally consistent pixel-wise object tracks over a vid…
Given a video and its incomplete textural description with missing words, the Video-Fill-in-the-Blank (ViFitB) task is to automatically find the missing word. The contextual information of the sentences are important to infer the missing words; the visual cues are even more crucial to get a more accurate inference. In this paper, we presents a new method which intuitively takes advantage of the structure of the sentences and employs merging LSTMs (to merge two LSTMs) to tackle the problem with embedded textural and visual cues. In the experiments, we have demonstrated the superior performance …
ReToMe-VA: Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）
Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）
Shanghai Jiao Tong University（上海交通大学）；Tencent AI Lab（腾讯人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；University of Michigan（密歇根大学）
In light of the success of contrastive learning in the image domain, current self-supervised video representation learning methods usually employ contrastive loss to facilitate video representation learning. When naively pulling two augmented views of a video closer, the model however tends to learn the common static background as a shortcut but fails to capture the motion information, a phenomenon dubbed as background bias. Such bias makes the model suffer from weak generalization ability, leading to worse performance on downstream tasks such as action recognition. To alleviate such bias, we …
We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tunin…
TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval
School of Software, Tsinghua University（清华大学软件学院）；BNRist, Tsinghua University（清华大学BNRist）；Hangzhou Zhuoxi Institute of Brain and Intelligence（杭州浙西脑智研究院）；GRG Banking Equipment Co., Ltd.（广电运通金融电子股份有限公司）；South China University of Technology（华南理工大学）
In this paper, the task of video panoptic segmentation is studied and two different methods to solve the task will be proposed. Video panoptic segmentation (VPS) is a recently introduced computer vision task that requires classifying and tracking every pixel in a given video. The nature of this task makes the cost of annotating datasets for it prohibiting. To understand video panoptic segmentation, first, earlier introduced constituent tasks that focus on semantics and tracking separately will be researched. Thereafter, two data-driven approaches which do not require training on a tailored VPS…
Selecting human objects out of the various type of objects in images and merging them with other scenes is manual and day-to-day work for photo editors. Although recently Adobe photoshop released "select subject" tool which automatically selects the foreground object in an image, but still requires fine manual tweaking separately. In this work, we proposed an application utilizing Mask R-CNN (for object detection and mask segmentation) that can extract human instances from multiple images and merge them with a new background. This application does not add any overhead to Mask R-CNN, running at…
Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videos
University of Oxford（牛津大学）；University of Technology Nuremberg（纽伦堡工业大学）
V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Shenzhen Loop Area Institute（深圳河套学院）；Sichuan University（四川大学）；EPIC Lab, Shanghai Jiao Tong University（上海交通大学EPIC实验室）
B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；International School of Information Science and Engineering, Dalian University of Technology（大连理工大学信息科学与工程国际学院）；Advanced Micro Devices（先进微器件公司）；School of Science, Edith Cowan University（埃迪斯科文大学科学学院）
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
University of Stuttgart（斯图加特大学）；DFKI；Max Planck Research School（马克斯·普朗克研究学校）；University of Oldenburg（奥尔登堡大学）；Technical University of Darmstadt（达姆施塔特技术大学）；VinRobotics；VinUniversity（Vin大学）；Aalto University（阿莱大学）；Queensland University of Technology（昆士兰技术大学）；HUST（华中科技大学）；Technical University of Denmark（丹麦技术大学）；University of Arkansas（阿肯色大学）；UC San Diego（圣地亚哥大学）；Stanford University（斯坦福大学）
