Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism while jointly attending to information in various representation subspaces, we construct our attention mechanism by means of an efficient convolution operation. Moreover, to alleviate the fragility of our Transformer model against adversarial attacks, we attempt to learn smoother decision boundaries. To this end, we augment the shape information of an image in the high-level feature space by permuting the feature mean and variance within mini-batches. With less computational complexity, our proposed hybrid model demonstrates its high robustness and generalization ability compared to the state-of-the-art studies on a large-scale collection of standardized MedMNIST-2D datasets.
Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
Department of Electrical and Computer Engineering, Concordia University, Montreal, Canada（电气与计算机工程系，康科迪亚大学，蒙特利尔，加拿大）；Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada（计算机科学与软件工程系，康科迪亚大学，蒙特利尔，加拿大）
Effect of Patch Size on Fine-Tuning Vision Transformers in Two-Dimensional and Three-Dimensional Medical Image Classification
Research Center for Medical Image Analysis and Artificial Intelligence（医学图像分析与人工智能研究中心）；Department of Medicine（医学系）；Faculty of Medicine and Dentistry（医学与牙科学院）；Danube Private University（多瑙私立大学）
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependenci…
Nanjing University（南京大学）；National Institute of Healthcare Data Science at Nanjing University（南京大学国家医疗健康数据科学研究院）；Istanbul Technical University（伊斯坦布尔理工大学）；University of Dundee（邓迪大学）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能有限公司）
Transformers have dominated the field of natural language processing, and recently impacted the computer vision area. In the field of medical image analysis, Transformers have also been successfully applied to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. Our paper aims to promote awareness and application of Transformers in the field of medical image analysis. Specifically, we first overview the core concepts of the attention mechanism built into Transformers and other basic components. Second, we review vario…
Chinese Academy of Sciences（中国科学院）；Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学院）；Vanderbilt University（范德堡大学）；University of Science and Technology of China（中国科学技术大学）
Transformer, the latest technological advance of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that cha…
Given the prevalence of 3D medical imaging technologies such as MRI and CT that are widely used in diagnosing and treating diverse diseases, 3D segmentation is one of the fundamental tasks of medical image analysis. Recently, Transformer-based models have started to achieve state-of-the-art performances across many vision tasks, through pre-training on large-scale natural image benchmark datasets. While works on medical image analysis have also begun to explore Transformer-based models, there is currently no optimal strategy to effectively leverage pre-trained Transformers, primarily due to th…
Imperial College London（帝国理工学院）；Technische Universität München（慕尼黑工业大学）；FAU Erlangen–Nürnberg（埃尔朗根-纽伦堡大学）；National Institutes of Health（美国国立卫生研究院）
Chest radiographs are one of the most common diagnostic modalities in clinical routine. It can be done cheaply, requires minimal equipment, and the image can be diagnosed by every radiologists. However, the number of chest radiographs obtained on a daily basis can easily overwhelm the available clinical capacities. We propose RATCHET: RAdiological Text Captioning for Human Examined Thoraces. RATCHET is a CNN-RNN-based medical transformer that is trained end-to-end. It is capable of extracting image features from chest radiographs, and generates medically accurate text reports that fit seamless…
Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review
University of Regensburg（雷根斯堡大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所）；RWTH Aachen University（亚琛工业大学）；Iran University of Science and Technology（伊朗科技大学）；Shahid Beheshti University（沙希德·贝赫什提大学）
Data Science Group, Faculty of Science Radboud University, Nijmegen, the Netherlands（拉德堡德大学理学院数据科学组）
Multi-view medical image analysis often depends on the combination of information from multiple views. However, differences in perspective or other forms of misalignment can make it difficult to combine views effectively, as registration is not always possible. Without registration, views can only be combined at a global feature level, by joining feature vectors after global pooling. We present a novel cross-view transformer method to transfer information between unregistered views at the level of spatial feature maps. We demonstrate this method on multi-view mammography and chest X-ray datase…
The Hong Kong University of Science and Technology（香港科技大学）
Affine registration is indispensable in a comprehensive medical image registration pipeline. However, only a few studies focus on fast and robust affine registration algorithms. Most of these studies utilize convolutional neural networks (CNNs) to learn joint affine and non-parametric registration, while the standalone performance of the affine subnetwork is less explored. Moreover, existing CNN-based affine registration approaches focus either on the local misalignment or the global orientation and position of the input to predict the affine transformation matrix, which are sensitive to spati…
University of Wollongong（伍伦贡大学）；Shenzhen International Graduate School of Tsinghua University（清华大学深圳国际研究生院）；University of Sydney（悉尼大学）
Automated radiographic report generation is a challenging cross-domain task that aims to automatically generate accurate and semantic-coherence reports to describe medical images. Despite the recent progress in this field, there are still many challenges at least in the following aspects. First, radiographic images are very similar to each other, and thus it is difficult to capture the fine-grained visual differences using CNN as the visual feature extractor like many existing methods. Further, semantic information has been widely applied to boost the performance of generation tasks (e.g. imag…
Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Ulm University Medical Center（乌尔姆大学医学中心）；BGZ - Bildgebungszentrum（BGZ影像中心）；i2SouI - Innovative Imaging in Surgical Oncology Ulm（乌尔姆外科肿瘤创新影像中心）；XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology（XAIRAD实验放射学人工智能合作中心）
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Vision Transformers (ViT) are competing to replace Convolutional Neural Networks (CNN) for various computer vision tasks in medical imaging such as classification and segmentation. While the vulnerability of CNNs to adversarial attacks is a well-known problem, recent works have shown that ViTs are also susceptible to such attacks and suffer significant performance degradation under attack. The vulnerability of ViTs to carefully engineered adversarial samples raises serious concerns about their safety in clinical settings. In this paper, we propose a novel self-ensembling method to enhance the …
Transformer, a model comprising attention-based encoder-decoder architecture, have gained prevalence in the field of natural language processing (NLP) and recently influenced the computer vision (CV) space. The similarities between computer vision and medical imaging, reviewed the question among researchers if the impact of transformers on computer vision be translated to medical imaging? In this paper, we attempt to provide a comprehensive and recent review on the application of transformers in medical imaging by; describing the transformer model comparing it with a diversity of convolutional…
Southeast University（东南大学）；Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing（江苏省医学信息处理联合国际研究实验室）；University of Western Ontario（西安大略大学）
An effective backbone network is important to deep learning-based Deformable Medical Image Registration (DMIR), because it extracts and matches the features between two images to discover the mutual correspondence for fine registration. However, the existing deep networks focus on single image situation and are limited in registration task which is performed on paired images. Therefore, we advance a novel backbone network, XMorpher, for the effective corresponding feature representation in DMIR. 1) It proposes a novel full transformer architecture including dual parallel feature extraction net…
Transformer has achieved impressive successes for various computer vision tasks. However, most of existing studies require to pretrain the Transformer backbone on a large-scale labeled dataset (e.g., ImageNet) for achieving satisfactory performance, which is usually unavailable for medical images. Additionally, due to the gap between medical and natural images, the improvement generated by the ImageNet pretrained weights significantly degrades while transferring the weights to medical image processing tasks. In this paper, we propose Bootstrap Own Latent of Transformer (BOLT), a self-supervise…
AstraZeneca（阿斯利康）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Science for Life Laboratory（生命科学实验室）
Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding similar levels of performance while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is time to move to transformer-based models or if we should keep working with CNNs - can we trivially switch to transformers? If so, what are the advantages and drawbacks of switching to ViTs for medical ima…
Zhejiang University（浙江大学）；University of Notre Dame（圣母大学）；Gachon University（嘉泉大学）；Shanghai University（上海大学）
Computer-aided medical image segmentation has been applied widely in diagnosis and treatment to obtain clinically useful information of shapes and volumes of target organs and tissues. In the past several years, convolutional neural network (CNN) based methods (e.g., U-Net) have dominated this area, but still suffered from inadequate long-range information capturing. Hence, recent work presented computer vision Transformer variants for medical image segmentation tasks and obtained promising performances. Such Transformers model long-range dependency by computing pair-wise patch relations. Howe…
MBZ University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Monash University（蒙纳士大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Agency for Science, Technology and Research（新加坡科技研究局）
Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as {de facto} operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of…
The Athenian School（雅典纳中学）；Cornell University（康奈尔大学）
Because of the necessity to obtain high-quality images with minimal radiation doses, such as in low-field magnetic resonance imaging, super-resolution reconstruction in medical imaging has become more popular (MRI). However, due to the complexity and high aesthetic requirements of medical imaging, image super-resolution reconstruction remains a difficult challenge. In this paper, we offer a deep learning-based strategy for reconstructing medical images from low resolutions utilizing Transformer and Generative Adversarial Networks (T-GAN). The integrated system can extract more precise texture …
Istanbul Technical University（伊斯坦布尔技术大学）；Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital（穆罕默德·阿基夫·埃尔索伊胸心血管外科培训研究医院）；Acibadem University（阿哲巴登大学）；King’s College London（伦敦国王学院）
To maintain a standard in a medical imaging study, images should have necessary image quality for potential diagnostic use. Although CNN-based approaches are used to assess the image quality, their performance can still be improved in terms of accuracy. In this work, we approach this problem by using Swin Transformer, which improves the poor-quality image classification performance that causes the degradation in medical image quality. We test our approach on Foreign Object Classification problem on Chest X-Rays (Object-CXR) and Left Ventricular Outflow Tract Classification problem on Cardiac M…
Bilkent University（比尔肯特大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心（UMRAM））；Sabuncu Brain Research Center（萨本库脑研究中心）
Generative adversarial models with convolutional neural network (CNN) backbones have recently been established as state-of-the-art in numerous medical image synthesis tasks. However, CNNs are designed to perform local processing with compact filters, and this inductive bias compromises learning of contextual features. Here, we propose a novel generative adversarial approach for medical image synthesis, ResViT, that leverages the contextual sensitivity of vision transformers along with the precision of convolution operators and realism of adversarial learning.} ResViT's generator employs a cent…
Transfer learning has gained attention in medical image analysis due to limited annotated 3D medical datasets for training data-driven deep learning models in the real world. Existing 3D-based methods have transferred the pre-trained models to downstream tasks, which achieved promising results with only a small number of training samples. However, they demand a massive amount of parameters to train the model for 3D medical imaging. In this work, we propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of …
Multi-Scale Transformer Architecture for Accurate Medical Image Classification
Tulane University（杜兰大学）；University of Houston（休斯顿大学）；University of Pennsylvania（宾夕法尼亚大学）；Shanghai Jiao Tong University（上海交通大学）；New York University（纽约大学）；New York University Shanghai（纽约大学上海分校）
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
Junia（朱尼亚学院）；Centrale Lille（里尔中央理工学院）；University of the Basque Country UPV/EHU（巴斯克大学 UPV/EHU）；Université Polytechnique Hauts-de-France（上法兰西理工大学）；Université de Lille（里尔大学）；Ho Chi Minh City Open University（胡志明市开放大学）
Nagoya Institute of Technology（名古屋工业大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Kurume University School of Medicine（久留米大学医学院）；Nagoya University（名古屋大学）
In medical image diagnosis, identifying the attention region, i.e., the region of interest for which the diagnosis is made, is an important task. Various methods have been developed to automatically identify target regions from given medical images. However, in actual medical practice, the diagnosis is made based not only on the images but also on a variety of clinical records. This means that pathologists examine medical images with some prior knowledge of the patients and that the attention regions may change depending on the clinical records. In this study, we propose a method called the Pe…
National University of Sciences and Technology (NUST)（国立科技大学（NUST））
Recent escalation in the field of computer vision underpins a huddle of algorithms with the magnificent potential to unravel the information contained within images. These computer vision algorithms are being practised in medical image analysis and are transfiguring the perception and interpretation of Imaging data. Among these algorithms, Vision Transformers are evolved as one of the most contemporary and dominant architectures that are being used in the field of computer vision. These are immensely utilized by a plenty of researchers to perform new as well as former experiments. Here, in thi…
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）；University of Electronic Science and Technology of China（电子科技大学）
Medical image segmentation is important for computer-aided diagnosis. Good segmentation demands the model to see the big picture and fine details simultaneously, i.e., to learn image features that incorporate large context while keep high spatial resolutions. To approach this goal, the most widely used methods -- U-Net and variants, extract and fuse multi-scale features. However, the fused features still have small "effective receptive fields" with a focus on local image cues, limiting their performance. In this work, we propose Segtran, an alternative segmentation framework based on transform…
Johns Hopkins Medical Institutes（约翰·霍普金斯医学院）；Johns Hopkins University（约翰·霍普金斯大学）；NVIDIA Corporation（英伟达公司）；Duke University Medical Center（杜克大学医学中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehensi…
Stable Vision Concept Transformers for Medical Diagnosis
Provable Responsible AI and Data Analytics (PRADA) Lab（可证责任AI与数据分析实验室）；King Abdullah University of Science and Technology（卡瓦尔大学科学与技术大学）；Tsinghua University（清华大学）；University of Auckland（奥克兰大学）
University of Science and Technology Beijing（北京科技大学）；University of Central Florida（中佛罗里达大学）
Transformer, benefiting from global (long-range) information modeling using self-attention mechanism, has been successful in natural language processing and computer vision recently. Convolutional Neural Networks, capable of capturing local features, are difficult to model explicit long-distance dependencies from global feature space. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we present the further attempt to exploit Transformer in 3D CNN for 3D medical image volumetric segmentation and propose a…
Convolution Neural Networks (CNNs) are widely used in medical image analysis, but their performance degrade when the magnification of testing images differ from the training images. The inability of CNNs to generalize across magnification scales can result in sub-optimal performance on external datasets. This study aims to evaluate the robustness of various deep learning architectures in the analysis of breast cancer histopathological images with varying magnification scales at training and testing stages. Here we explore and compare the performance of multiple deep learning architectures, inc…
Philips Innovation Labs Rus（飞利浦俄罗斯创新实验室）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Sber AI Lab（俄联邦储蓄银行人工智能实验室）
The automatic clinical caption generation problem is referred to as proposed model combining the analysis of frontal chest X-Ray scans with structured patient information from the radiology records. We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed combination of these models generates a textual summary with the essential information about pathologies found, their location, and the 2D heatmaps localizing each pathology on the original X-Ray scans. The proposed model is tested on two medical datasets, th…
Compute-Efficient Medical Image Classification with Softmax-Free Transformers and Sequence Normalization
University Hospital Aachen（亚琛大学医院）；Else Kroener Fresenius Center for Digital Health（埃尔瑟·克勒纳·弗雷森纽斯数字健康中心）；Medical Faculty Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯医学院）；Technical University Dresden（德累斯顿工业大学）；University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯大学医院及医学院）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；University Hospital Heidelberg（海德堡大学医院）
Vision Transformers (ViT)s have shown great performance in self-supervised learning of global and local representations that can be transferred to downstream applications. Inspired by these results, we introduce a novel self-supervised learning framework with tailored proxy tasks for medical image analysis. Specifically, we propose: (i) a new 3D transformer-based model, dubbed Swin UNEt TRansformers (Swin UNETR), with a hierarchical encoder for self-supervised pre-training; (ii) tailored proxy tasks for learning the underlying pattern of human anatomy. We demonstrate successful pre-training of…
Combining information from multi-view images is crucial to improve the performance and robustness of automated methods for disease diagnosis. However, due to the non-alignment characteristics of multi-view images, building correlation and data fusion across views largely remain an open problem. In this study, we present TransFusion, a Transformer-based architecture to merge divergent multi-view imaging information using convolutional layers and powerful attention mechanisms. In particular, the Divergent Fusion Attention (DiFA) module is proposed for rich cross-view context modeling and semanti…
From CNN to Transformer: A Review of Medical Image Segmentation Models
University of Electronic Science and Technology of China（电子科技大学）；Deyang People’s Hospital（德阳市人民医院）；Sichuan Provincial People’s Hospital（四川省人民医院）；Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital（中国科学院四川转化医学研究医院）
