This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias of locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a token). We showcase the effectiveness of pixels-as-tokens across three well-studied computer vision tasks: supervised learning for classification and regression, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although it's computationally less practical to directly operate on individual pixels, we believe the community must be made aware of this surprising piece of knowledge when devising the next generation of neural network architectures for computer vision.
Guangdong University of Technology（广东工业大学）；The University of Sydney（悉尼大学）
Unpaired image-to-image translation aims to find a mapping between the source domain and the target domain. To alleviate the problem of the lack of supervised labels for the source images, cycle-consistency based methods have been proposed for image structure preservation by assuming a reversible relationship between unpaired images. However, this assumption only uses limited correspondence between image pairs. Recently, contrastive learning (CL) has been used to further investigate the image correspondence in unpaired image translation by using patch-based positive/negative learning. Patch-ba…
Exploring the Coordination of Frequency and Attention in Masked Image Modeling
School of Cyber Science and Engineering, Southeast University（东南大学网络空间科学与工程学院）；Purple Mountain Laboratories（紫金山实验室）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Alibaba Group（阿里巴巴集团）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）；Zhuhai UM Science and Technology Research Institute（珠海澳大科技研究院）；Faculty of Science & Technology at University of Macau（澳门大学科技学院）
Tsinghua University（清华大学）；Renmin University of China（中国人民大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and class-conditional image generation, as well as text-to-image generation tasks, where U-ViT is compara…
Recently, contrastive learning-based image translation methods have been proposed, which contrasts different spatial locations to enhance the spatial correspondence. However, the methods often ignore the diverse semantic relation within the images. To address this, here we propose a novel semantic relation consistency (SRC) regularization along with the decoupled contrastive learning, which utilize the diverse semantics by focusing on the heterogeneous semantics between the image patches of a single image. To further improve the performance, we present a hard negative mining by exploiting the …
Exploring Local Memorization in Diffusion Models via Bright Ending Attention
School of Computer Science, Faculty of Engineering, The University of Sydney, Australia（悉尼大学计算机科学学院、工程学院）；School of Physics, Mathematics and Computing, The University of Western Australia, Australia（西澳大学物理、数学与计算学院）；Center for Research in Computer Vision, University of Central Florida, USA（佛罗里达中央大学计算机视觉研究中心）
This paper proposes a content relationship distillation (CRD) to tackle the over-parameterized generative adversarial networks (GANs) for the serviceability in cutting-edge devices. In contrast to traditional instance-level distillation, we design a novel GAN compression oriented knowledge by slicing the contents of teacher outputs into multiple fine-grained granularities, such as row/column strips (global information) and image patches (local information), modeling the relationships among them, such as pairwise distance and triplet-wise angle, and encouraging the student to capture these rela…
Frankfurt Institute for Advanced Studies（法兰克福高等研究院）；Xidian FIAS Joint Research Center（西电-法兰克福高等研究院联合研究中心）
The vision transformer (ViT) has achieved state-of-the-art results in various vision tasks. It utilizes a learnable position embedding (PE) mechanism to encode the location of each image patch. However, it is presently unclear if this learnable PE is really necessary and what its benefits are. This paper explores two alternative ways of encoding the location of individual patches that exploit prior knowledge about their spatial arrangement. One is called the sequence relationship embedding (SRE), and the other is called the circle relationship embedding (CRE). Among them, the SRE considers all…
The University of Adelaide（阿德莱德大学）；Rutgers University（罗格斯大学）
Deep neural networks are vulnerable to attacks from adversarial inputs and, more recently, Trojans to misguide or hijack the model's decision. We expose the existence of an intriguing class of spatially bounded, physically realizable, adversarial examples -- Universal NaTuralistic adversarial paTches -- we call TnTs, by exploring the superset of the spatially bounded adversarial example space and the natural input space within generative adversarial networks. Now, an adversary can arm themselves with a patch that is naturalistic, less malicious-looking, physically realizable, highly effective …
University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；GE Global Research Center（通用电气全球研究中心）
Recent works succeeded to generate adversarial perturbations on the entire image or the object of interests to corrupt CNN based object detectors. In this paper, we focus on exploring the vulnerability of the Single Shot Module (SSM) commonly used in recent object detectors, by adding small perturbations to patches in the background outside the object. The SSM is referred to the Region Proposal Network used in a two-stage object detector or the single-stage object detector itself. The SSM is typically a fully convolutional neural network which generates output in a single forward pass. Due to …
Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；Huazhong University of Science and Technology（华中科技大学）；Zhejiang University（浙江大学）；Beijing Institute of Technology（北京理工大学）
We launch EVA, a vision-centric foundation model to explore the limits of visual representation at scale using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreo…
Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement
Nanyang Technological University（南洋理工大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Thapar University（塔帕尔大学）；L3i, University of La-Rochelle（拉罗谢尔大学L3i实验室）
In this work, we adhere to explore a Multi-Tasking learning (MTL) based network to perform document attribute classification such as the font type, font size, font emphasis and scanning resolution classification of a document image. To accomplish these tasks, we operate on either segmented word level or on uniformed size patches randomly cropped out of the document. Furthermore, a hybrid convolution neural network (CNN) architecture "MTL+MI", which is based on the combination of MTL and Multi-Instance (MI) of patch and word is used to accomplish joint learning for the classification of the sam…
The Chinese University of Hong Kong（香港中文大学）；SmartMore
We study the vision transformer structure in the mobile level in this paper, and find a dramatic performance drop. We analyze the reason behind this phenomenon, and propose a novel irregular patch embedding module and adaptive patch fusion module to improve the performance. We conjecture that the vision transformer blocks (which consist of multi-head attention and feed-forward network) are more suitable to handle high-level information than low-level features. The irregular patch embedding module extracts patches that contain rich high-level information with different receptive fields. The tra…
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple m…
Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
IDEAS NCBR；Jagiellonian University, Faculty of Mathematics and Computer Science（雅盖隆大学数学与计算机科学学院）；Jagiellonian University, Doctoral School of Exact and Natural Sciences（雅盖隆大学精确与自然科学博士学院）；University of Warsaw（华沙大学）；Warsaw University of Technology（华沙技术大学）；Tooploox
Electronics and Telecommunications Research Institute (ETRI)（电子与电信研究所）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；AITRICS
Masked image modeling (MIM) has become a popular strategy for self-supervised learning~(SSL) of visual representations with Vision Transformers. A representative MIM model, the masked auto-encoder (MAE), randomly masks a subset of image patches and reconstructs the masked patches given the unmasked patches. Concurrently, many recent works in self-supervised learning utilize the student/teacher paradigm which provides the student with an additional target based on the output of a teacher composed of an exponential moving average (EMA) of previous students. Although common, relatively little is …
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
ISCAS（中国科学院软件研究所）；UCAS（中国科学院大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Peking University（北京大学）；University of Macau（澳门大学）
Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new Vision GNN (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation …
Tsinghua University（清华大学）；Huawei Technologies Ltd.（华为技术有限公司）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individ…
South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）；Ant Group（蚂蚁集团）
Recently self-supervised representation learning has drawn considerable attention from the scene text recognition community. Different from previous studies using contrastive learning, we tackle the issue from an alternative perspective, i.e., by formulating the representation learning scheme in a generative manner. Typically, the neighboring image patches among one text line tend to have similar styles, including the strokes, textures, colors, etc. Motivated by this common sense, we augment one image patch and use its neighboring patch as guidance to recover itself. Specifically, we propose a…
Institute of Software Chinese Academy of Science（中国科学院软件研究所）；Institute of Computing Technology Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Although vision Transformers have achieved excellent performance as backbone models in many vision tasks, most of them intend to capture global relations of all tokens in an image or a window, which disrupts the inherent spatial and local correlations between patches in 2D structure. In this paper, we introduce a simple vision Transformer named SimViT, to incorporate spatial structure and local information into the vision Transformers. Specifically, we introduce Multi-head Central Self-Attention(MCSA) instead of conventional Multi-head Self-Attention to capture highly local relations. The intr…
Exploring Token-Level Augmentation in Vision Transformer for Semi-Supervised Semantic Segmentation
South China University of Technology（华南理工大学）；Pengcheng Laboratory（鹏城实验室）；China Tobacco Guangdong Industrial Corporation（广东中烟工业有限责任公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Huaqiao University（华侨大学）；Nanjing University of Science and Technology（南京理工大学）；University of Science and Technology Beijing（北京科技大学）
Transformers are more and more popular in computer vision, which treat an image as a sequence of patches and learn robust global features from the sequence. However, pure transformers are not entirely suitable for vehicle re-identification because vehicle re-identification requires both robust global features and discriminative local features. For that, a graph interactive transformer (GiT) is proposed in this paper. In the macro view, a list of GiT blocks are stacked to build a vehicle re-identification model, in where graphs are to extract discriminative local features within patches and tra…
Nanyang Technological University（南洋理工大学）；The University of Adelaide（阿德莱德大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）
State-of-the-art semantic image segmentation methods are mostly based on training deep convolutional neural networks (CNNs). In this work, we proffer to improve semantic segmentation with the use of contextual information. In particular, we explore `patch-patch' context and `patch-background' context in deep CNNs. We formulate deep structured models by combining CNNs and Conditional Random Fields (CRFs) for learning the patch-patch context between image regions. Specifically, we formulate CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Effic…
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Tsinghua University（清华大学）；Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Technology Sydney（悉尼科技大学）；University of Oxford（牛津大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；DarkMatter AI Research（DarkMatter AI 研究院）
We address the problem of Visual Question Answering (VQA), which requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on convolutional-recurrent networks to this problem, but have failed to model spatial inference. To remedy this, we propose a model we call the Spatial Memory Network and apply it to the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. Our Spatial Memory Network st…
University of Electronic Science and Technology of China（电子科技大学）；Southwestern University of Finance and Economics（西南财经大学）
The convolution operation is a powerful tool for feature extraction and plays a prominent role in the field of computer vision. However, when targeting the pixel-wise tasks like image fusion, it would not fully perceive the particularity of each pixel in the image if the uniform convolution kernel is used on different patches. In this paper, we propose a local adaptive convolution (LAConv), which is dynamically adjusted to different spatial locations. LAConv enables the network to pay attention to every specific local area in the learning process. Besides, the dynamic bias (DYB) is introduced …
Exploring Multi-Timestep Multi-Stage Diffusion Features for Hyperspectral Image Classification
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Indian Institute of Information Technology Allahabad（阿拉哈巴德印度信息技术学院）；National University of Singapore（新加坡国立大学）；Bioinformatics Institute, A*STAR（新加坡科技研究局生物信息学研究所）；Image and Pervasive Access Lab(IPAL), CNRS UMI 2955（法国国家科学研究中心UMI 2955图像与普适访问实验室）；Singapore Eye Research Institute（新加坡眼科研究所）
Researchers working on computational analysis of Whole Slide Images (WSIs) in histopathology have primarily resorted to patch-based modelling due to large resolution of each WSI. The large resolution makes WSIs infeasible to be fed directly into the machine learning models due to computational constraints. However, due to patch-based analysis, most of the current methods fail to exploit the underlying spatial relationship among the patches. In our work, we have tried to integrate this relationship along with feature-based correlation among the extracted patches from the particular tumorous reg…
Exploring Patient Data Requirements in Training Effective AI Models for MRI-based Breast Cancer Classification
Center for Biosystems and Biotech Data Science, Ghent University Global Campus（根特大学全球校区生物系统与生物科技数据科学中心）；The State University of New York Korea（韩国州立纽约大学）；Ghent University（根特大学）
Benefiting from Quantum? A Comparative Study of Q-Seg, Quantum-Inspired Techniques, and U-Net for Crack Segmentation
Fraunhofer Institute for Industrial Mathematics (ITWM)（弗劳恩霍夫工业数学研究所）；RPTU Kaiserslautern-Landau（凯撒斯劳滕-兰道莱茵-普法尔茨理工大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；Shanghai Jiao Tong University（上海交通大学）
Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable compl…
Exploring shared memory architectures for end-to-end gigapixel deep learning
Vanderbilt University（范德堡大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；National Institutes of Health（美国国立卫生研究院）；Vanderbilt University Medical Center（范德堡大学医学中心）；Veterans Affairs Tennessee Valley Healthcare System（退伍军人事务部田纳西谷医疗系统）
