The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.
MBZ University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);Monash University(蒙纳士大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);Agency for Science, Technology and Research(新加坡科技研究局)
Shifted Windows Transformers for Medical Image Quality Assessment
Istanbul Technical University(伊斯坦布尔技术大学);Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital(穆罕默德·阿基夫·埃尔索伊胸心血管外科培训研究医院);Acibadem University(阿哲巴登大学);King’s College London(伦敦国王学院)
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
Junia(朱尼亚学院);Centrale Lille(里尔中央理工学院);University of the Basque Country UPV/EHU(巴斯克大学 UPV/EHU);Université Polytechnique Hauts-de-France(上法兰西理工大学);Université de Lille(里尔大学);Ho Chi Minh City Open University(胡志明市开放大学)
Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records
Nagoya Institute of Technology(名古屋工业大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Kurume University School of Medicine(久留米大学医学院);Nagoya University(名古屋大学)
Nanjing University(南京大学);National Institute of Healthcare Data Science at Nanjing University(南京大学国家医疗健康数据科学研究院);Istanbul Technical University(伊斯坦布尔理工大学);University of Dundee(邓迪大学);ShanghaiTech University(上海科技大学);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能有限公司)
Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives
Chinese Academy of Sciences(中国科学院);Johns Hopkins Medical Institutes(约翰斯·霍普金斯医学院);Vanderbilt University(范德堡大学);University of Science and Technology of China(中国科学技术大学)
TransMorph: Transformer for unsupervised medical image registration
Johns Hopkins Medical Institutes(约翰·霍普金斯医学院);Johns Hopkins University(约翰·霍普金斯大学);NVIDIA Corporation(英伟达公司);Duke University Medical Center(杜克大学医学中心);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
From CNN to Transformer: A Review of Medical Image Segmentation Models
University of Electronic Science and Technology of China(电子科技大学);Deyang People’s Hospital(德阳市人民医院);Sichuan Provincial People’s Hospital(四川省人民医院);Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital(中国科学院四川转化医学研究医院)
MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation
German Cancer Research Center (DKFZ)(德国癌症研究中心);Heidelberg University(海德堡大学);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心);University Medical Center Heidelberg(海德堡大学医院);Helmholtz Imaging(亥姆霍兹成像机构)
Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review
University of Regensburg(雷根斯堡大学);Fraunhofer Institute for Digital Medicine MEVIS(弗劳恩霍夫数字医学研究所);RWTH Aachen University(亚琛工业大学);Iran University of Science and Technology(伊朗科技大学);Shahid Beheshti University(沙希德·贝赫什提大学)
Investigating the Robustness of Vision Transformers against Label Noise in Medical Image Classification
Center for Imaging Science, Rochester Institute of Technology(罗切斯特理工大学成像科学中心);NepAl Applied Mathematics and Informatics Institute for Research (NAAMII)(尼泊尔应用数学与信息学研究所);University of Aberdeen(阿伯丁大学);Biomedical Engineering, Rochester Institute of Technology(罗切斯特理工大学生物医学工程系)
Multi-Scale Transformer Architecture for Accurate Medical Image Classification
Tulane University(杜兰大学);University of Houston(休斯顿大学);University of Pennsylvania(宾夕法尼亚大学);Shanghai Jiao Tong University(上海交通大学);New York University(纽约大学);New York University Shanghai(纽约大学上海分校)
Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Ulm University Medical Center(乌尔姆大学医学中心);BGZ - Bildgebungszentrum(BGZ影像中心);i2SouI - Innovative Imaging in Surgical Oncology Ulm(乌尔姆外科肿瘤创新影像中心);XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology(XAIRAD实验放射学人工智能合作中心)
Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
Department of Electrical and Computer Engineering, Concordia University, Montreal, Canada(电气与计算机工程系,康科迪亚大学,蒙特利尔,加拿大);Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada(计算机科学与软件工程系,康科迪亚大学,蒙特利尔,加拿大)
EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis
School of Computing, University of Georgia(佐治亚大学计算学院);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Georgia(佐治亚大学);Department of Physics and Astronomy(物理与天文学系)
XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention
Southeast University(东南大学);Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing(江苏省医学信息处理联合国际研究实验室);University of Western Ontario(西安大略大学)
有效的骨干网络对于基于深度学习的形变医学图像配准(Deformable Medical Image Registration,DMIR)非常重要,因为它可以提取和匹配两幅图像之间的特征,以发现相互之间的对应关系,从而实现精细配准。然而,现有的深度网络集中在单个图像的情况下,并且在配对图像上执行的配准任务受到限制。因此,我们提出了一种新的骨干网络,XMorpher,在DMIR有效的对应特征表示。1)提出了一种新的完整的Transformer架构,包括双并行特征提取网络,通过交叉注意交换信息,从而发现多层次的语义对应,同时逐渐提取各个特征,最终实现有效的配准。2)提出了交叉注意力Transformer(CAT)模块,建立图像间的注意力机制,能够自动发现对应关系,促使特征在网络中有效融合。3)该算法将注意力计算限制在不同大小的基窗口和搜索窗口之间,从而集中在变形配准的局部变换上,同时提高了计算效率。在没有任何额外功能的情况下,我们的XMorpher在DSC上提供了Voxelmorph 2.8%的改进,证明了它在DMIR中有效地表示了配对图像的特征。我们相信我们的XMorpher在更多配对的医学图像中具有巨大的应用潜力。我们的XMorpher在https://github.com/Solemoon/XMorpher上开放
Stable Vision Concept Transformers for Medical Diagnosis
Provable Responsible AI and Data Analytics (PRADA) Lab(可证责任AI与数据分析实验室);King Abdullah University of Science and Technology(卡瓦尔大学科学与技术大学);Tsinghua University(清华大学);University of Auckland(奥克兰大学)