Non-overlapping patch-wise convolution is the default image tokenizer for all state-of-the-art vision Transformer (ViT) models. Even though many ViT variants have been proposed to improve its efficiency and accuracy, little research on improving the image tokenizer itself has been reported in the literature. In this paper, we propose a new image tokenizer based on wavelet transformation. We show that ViT models with the new tokenizer achieve both higher training throughput and better top-1 precision for the ImageNet validation set. We present a theoretical analysis on why the proposed tokenizer improves the training throughput without any change to ViT model architecture. Our analysis suggests that the new tokenizer can effectively handle high-resolution images and is naturally resistant to adversarial attack. Furthermore, the proposed image tokenizer offers a fresh perspective on important new research directions for ViT-based model design, such as image tokens on a non-uniform grid for image understanding.
The Chinese University of Hong Kong（香港中文大学）；Amazon AI（亚马逊人工智能部门）
The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm. Relying on the tokenization process that splits inputs into multiple tokens, transformers are capable of extracting their pairwise relationships using self-attention. While being the stemming building block of transformers, what makes for a good tokenizer has not been well understood in computer vision. In this work, we investigate this uncharted problem from an information trade-off perspective. In addition to unifying and understanding exis…
National University of Singapore（新加坡国立大学）；Tencent PCG（腾讯平台与内容事业群）
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i.e., VQ tokenizers and generative transformers. Prior research focuses on improving the reconstruction fidelity of VQ tokenizers but rarely examines how the improvement in reconstruction affects the generation ability of generative transformers. In this paper, we surprisingly find that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation. Instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability …
Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformers
Predictive Analytics Lab, University of Sussex, UK（苏塞克斯大学预测分析实验室）；University of Surrey, UK（塞夫尔大学）；Basque Center for Applied Mathematics, Spain（巴斯克应用数学中心）；Monash University, Indonesia（墨尔本大学）
Currently, many face forgery detection methods aggregate spatial and frequency features to enhance the generalization ability and gain promising performance under the cross-dataset scenario. However, these methods only leverage one level frequency information which limits their expressive ability. To overcome these limitations, we propose a multi-scale wavelet transformer framework for face forgery detection. Specifically, to take full advantage of the multi-scale and multi-frequency wavelet representation, we gradually aggregate the multi-scale wavelet representation at different stages of th…
EWT: Efficient Wavelet-Transformer for Single Image Denoising
Shanghai University（上海大学）；Nanjing University of Science and Technology（南京理工大学）；Xidian University（西安电子科技大学）；Beijing Electronic Science & Technology Institute（北京电子科技学院）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Soochow University（苏州大学）；The Chinese University of Hong Kong（香港中文大学）
Language-Guided Transformer Tokenizer for Human Motion Generation
Transsion Ltd.（Transsion有限公司）；Chongqing University of Technology（重庆理工大学）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；Shenzhen Graduate School（深圳研究生院）
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）；Sun Yat-sen University（中山大学）；Nanjing University of Science and Technology（南京理工大学）
Real-world image denoising is a practical image restoration problem that aims to obtain clean images from in-the-wild noisy inputs. Recently, the Vision Transformer (ViT) has exhibited a strong ability to capture long-range dependencies, and many researchers have attempted to apply the ViT to image denoising tasks. However, a real-world image is an isolated frame that makes the ViT build long-range dependencies based on the internal patches, which divides images into patches, disarranges noise patterns and damages gradient continuity. In this article, we propose to resolve this issue by using …
ByteDance（字节跳动）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）；UC Santa Cruz（加州大学圣克鲁兹分校）
The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class t…
Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea（韩国科学技术院金载哲人工智能研究生院，大田，韩国）；School of Computing, KAIST, Daejeon, South Korea（韩国科学技术院计算学院，大田，韩国）
Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation
National Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（北京大学深圳研究生院通用人工智能国家重点实验室）；Amazon Prime Video（亚马逊Prime视频）；University of Trento（特伦托大学）
University of Macau（澳门大学）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）
Image enhancement is a technique that frequently utilized in digital image processing. In recent years, the popularity of learning-based techniques for enhancing the aesthetic performance of photographs has increased. However, the majority of current works do not optimize an image from different frequency domains and typically focus on either pixel-level or global-level enhancements. In this paper, we propose a transformer-based model in the wavelet domain to refine different frequency bands of an image. Our method focuses both on local details and high-level features for enhancement, which ca…
Multi-scale Vision Transformer (ViT) has emerged as a powerful backbone for computer vision tasks, while the self-attention computation in Transformer scales quadratically w.r.t. the input patch number. Thus, existing solutions commonly employ down-sampling operations (e.g., average pooling) over keys/values to dramatically reduce the computational cost. In this work, we argue that such over-aggressive down-sampling design is not invertible and inevitably causes information dropping especially for high-frequency components in objects (e.g., texture details). Motivated by the wavelet theory, we…
University of Chinese Academy of Sciences（中国科学院大学）；Microsoft Research（微软研究院）
Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a co…
Uncertainty-Aware Source-Free Adaptive Image Super-Resolution with Wavelet Augmentation Transformer
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；OPPO Research Institute（OPPO研究院）；The Hong Kong Polytechnic University（香港理工大学）；ShanghaiTech University（上海科技大学）
Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers
RWTH Aachen University（亚琛工业大学）；Iran University of Science and Technology（伊朗科技大学）；Universiti Kebangsaan（马来西亚国民大学）；University of Regensburg（雷根斯堡大学）；Shahid Beheshti University（沙希德·贝赫什提大学）
H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（国家一般人工智能实验室，北京大学深圳研究生院）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）；Amazon AGI（亚马逊人工智能实验室）；University of Trento（特伦托大学）
ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression
Electrical and Computer Engineering Department, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；School of Electronic Science and Technology, Eastern Institute of Technology（电子科学与技术学院，东部技术学院）；Google LLC（谷歌公司）
WaveFormer: A 3D Transformer with Wavelet-Driven Feature Representation for Efficient Medical Image Segmentation
University of Florida（佛罗里达大学）；University of Central Florida（中佛罗里达大学）；University of California Santa Cruz（加州大学圣克鲁兹分校）；Microsoft Healthcare AI（微软医疗人工智能）；AdventHealth（AdventHealth（ AdventHealth为美国大型非营利医疗保健系统，保留原名称））
WD-DETR: Wavelet Denoising-Enhanced Real-Time Object Detection Transformer for Robot Perception with Event Cameras
School of Aeronautic Science and Engineering, Beihang University（航空科学与工程学院，北京航空航天大学）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）；Institute of Unmanned System, Beihang University（无人系统研究院，北京航空航天大学）；Tianmushan Laboratory Xixi Octagon City（天门山实验室西溪八角城）
针对事件相机密集表示噪声导致漏检的问题，提出 WD-DETR，通过小波去噪骨干、Transformer 检测头和 DRCB 混合编码器提升精度，并在 Jetson Orin NX 上实现约35 FPS实时推理。
