Transformers have recently gained significant attention in the computer vision community. However, the lack of scalability of self-attention mechanisms with respect to image size has limited their wide adoption in state-of-the-art vision backbones. In this paper we introduce an efficient and scalable attention model we call multi-axis attention, which consists of two aspects: blocked local and dilated global attention. These design choices allow global-local spatial interactions on arbitrary input resolutions with only linear complexity. We also present a new architectural element by effectively blending our proposed attention model with convolutions, and accordingly propose a simple hierarchical vision backbone, dubbed MaxViT, by simply repeating the basic building block over multiple stages. Notably, MaxViT is able to ''see'' globally throughout the entire network, even in earlier, high-resolution stages. We demonstrate the effectiveness of our model on a broad spectrum of vision tasks. On image classification, MaxViT achieves state-of-the-art performance under various settings: without extra data, MaxViT attains 86.5% ImageNet-1K top-1 accuracy; with ImageNet-21K pre-training, our model achieves 88.7% top-1 accuracy. For downstream tasks, MaxViT as a backbone delivers favorable performance on object detection as well as visual aesthetic assessment. We also show that our proposed model expresses strong generative modeling capability on ImageNet, demonstrating the superior potential of MaxViT blocks as a universal vision module. The source code and trained models will be available at https://github.com/google-research/maxvit.
MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation
Pakistan Institute of Engineering & Applied Sciences（巴基斯坦工程与应用科学学院）；PIEAS Artificial Intelligence Center (PAIC)（PIEAS人工智能中心）；Center for Mathematical Sciences（数学科学中心）
CAD-RADS scoring of coronary CT angiography with Multi-Axis Vision Transformer: a clinically-inspired deep learning pipeline
University of Pavia（帕维亚大学）；Centro Cardiologico Monzino IRCCS（蒙齐诺心脏病学中心IRCCS）；University of Milan（米兰大学）；IRCCS Istituti Clinici Scientifici Maugeri（IRCCS莫杰里科学临床研究所）
Recent advances in transformer-based models have drawn attention to exploring these techniques in medical image segmentation, especially in conjunction with the U-Net model (or its variants), which has shown great success in medical image segmentation, under both 2D and 3D settings. Current 2D based methods either directly replace convolutional layers with pure transformers or consider a transformer as an additional intermediate encoder between the encoder and decoder of U-Net. However, these approaches only consider the attention encoding within one single slice and do not utilize the axial-a…
Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Krembil Research Institute, University Health Network（肯贝尔研究所，大学健康网络）
Recently, a variety of vision transformers have been developed as their capability of modeling long-range dependency. In current transformer-based backbones for medical image segmentation, convolutional layers were replaced with pure transformers, or transformers were added to the deepest encoder to learn global context. However, there are mainly two challenges in a scale-wise perspective: (1) intra-scale problem: the existing methods lacked in extracting local-global cues in each scale, which may impact the signal propagation of small objects; (2) inter-scale problem: the existing methods fai…
Columbia University（哥伦比亚大学）；Columbia University School of Professional Studies（哥伦比亚大学专业研究学院）；Department of Statistics（统计学系）；Columbia University Department of Computer Science（哥伦比亚大学计算机科学系）
Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems
University of Calgary（卡尔加里大学）；Hotchkiss Brain Institute, University of Calgary（卡尔加里大学霍奇基斯脑研究所）；Ulster University（阿尔斯特大学）；Intelligent Systems Research Center, Ulster University（阿尔斯特大学智能系统研究中心）
Mamba, a State Space Model (SSM), has recently shown competitive performance to Convolutional Neural Networks (CNNs) and Transformers in Natural Language Processing and general sequence modeling. Various attempts have been made to adapt Mamba to Computer Vision tasks, including medical image segmentation (MIS). Vision Mamba (VM)-based networks are particularly attractive due to their ability to achieve global receptive fields, similar to Vision Transformers, while also maintaining linear complexity in the number of tokens. However, the existing VM models still struggle to maintain both spatial…
We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner. Different video segmentation tasks (e.g., video semantic/instance/panoptic segmentation) are usually considered as distinct problems. State-of-the-art models adopted in the separate communities have diverged, and radically different approaches dominate in each task. By contrast, we make a crucial observation that video segmentation tasks could be generally formulated as the problem of assigning different predicted labels to video tubes (where a tube is obtained by linking segm…
Recently, Visual Transformer (ViT) has been widely used in various fields of computer vision due to applying self-attention mechanism in the spatial domain to modeling global knowledge. Especially in medical image segmentation (MIS), many works are devoted to combining ViT and CNN, and even some works directly utilize pure ViT-based models. However, recent works improved models in the aspect of spatial domain while ignoring the importance of frequency domain information. Therefore, we propose Multi-axis External Weights UNet (MEW-UNet) for MIS based on the U-shape architecture by replacing sel…
Learning Multi-axis Representation in Frequency Domain for Medical Image Segmentation
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
Evaluation of Vision Transformers for Multimodal Image Classification: A Case Study on Brain, Lung, and Kidney Tumors
Centro de Tecnologías de la Imagen (CTIM)（图像技术中心）；Instituto Universitario de Cibernética, Empresas y Sociedad (IUCES)（大学信息学与企业与社会研究所）；University of Las Palmas de Gran Canaria（拉斯帕尔马斯德格拉纳达大学）
CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection
Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada（计算机科学与软件工程系，康科迪亚大学，加拿大蒙特利尔）；Ebovir Biotechnologie Inc., Montreal, Canada（Ebovir生物技术公司，加拿大蒙特利尔）
MM-SurvNet: Deep Learning-Based Survival Risk Stratification in Breast Cancer Through Multimodal Data Fusion
School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；NSW Health Pathology, St. George Hospital（新南威尔士州卫生病理处圣乔治医院）
3D object detection in autonomous driving aims to reason "what" and "where" the objects of interest present in a 3D world. Following the conventional wisdom of previous 2D object detection, existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis. However, we conjugate that this does not fit the nature of the ego car's perspective, as each onboard camera perceives the world in shape of wedge intrinsic to the imaging geometry with radical (non-perpendicular) axis. Hence, in this paper we advocate the exploitation of the Polar coordinate system and propose a…
University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Following the successful application of vision transformers in multiple computer vision tasks, these models have drawn the attention of the signal processing community. This is because signals are often represented as spectrograms (e.g. through Discrete Fourier Transform) which can be directly provided as input to vision transformers. However, naively applying transformers to spectrograms is suboptimal. Since the axes represent distinct dimensions, i.e. frequency and time, we argue that a better approach is to separate the attention dedicated to each axis. To this end, we propose the Separable…
HiLRP: Toward One Trustworthy Explanation for Vision Transformer: Conservation-Valid Attribution via Attention Primitives
Faculty of Engineering, University of Peradeniya（佩拉德尼亚大学工程学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）；School of Engineering & Digital Technologies, University of Southern Queensland（南昆士兰大学工程与数字技术学院）；School of Engineering & Technologies, UNSW（新南威尔士大学工程与技术学院）；Faculty of Science and Technology, Charles Darwin University（查尔斯达尔文大学科学技术学院）
HMANet: Hybrid Multi-Axis Aggregation Network for Image Super-Resolution
College of Computer Science and Engineering, Shandong University of Science and Technology（山东科技大学计算机科学与工程学院）；School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）；School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；School of Electronic and Information Engineering, Harbin Institute of Technology（哈尔滨工业大学电子与信息工程学院）
本文提出 HMANet，通过堆叠 RHTB 与 GAB 扩展注意力交互范围，并结合新型预训练策略提升图像超分辨率性能，实验表明其在基准数据集上优于现有方法。
