Transformers have made significant strides across various artificial intelligence domains, including natural language processing, computer vision, and audio processing. This success has naturally garnered considerable interest from both academic and industry researchers. Consequently, numerous Transformer variants (often referred to as X-formers) have been developed for these fields. However, a thorough and systematic review of these modality-specific conversions remains lacking. Modality Conversion involves the transformation of data from one form of representation to another, mimicking the way humans integrate and interpret sensory information. This paper provides a comprehensive review of transformer-based models applied to the primary modalities of text, vision, and speech, discussing their architectures, conversion methodologies, and applications. By synthesizing the literature on modality conversion, this survey aims to underline the versatility and scalability of transformers in advancing AI-driven content generation and understanding.
NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation
Tianjin University(天津大学);The University of Hong Kong(香港大学);Huanhu Hospital(环湖医院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Slide-free MUSE Microscopy to H&E Histology Modality Conversion via Unpaired Image-to-Image Translation GAN Models
University of California, Davis(加州大学戴维斯分校);University of San Francisco Data Institute(旧金山大学数据研究所);Wicklow AI in Medicine Research Initiative(威克洛医学人工智能研究计划);University of Texas, San Antonio Health(德克萨斯大学圣安东尼奥健康分校)
Multi-Modal Transformer for Accelerated MR Imaging
Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Northwestern Polytechnical University(西北工业大学);Terminus Group(特斯联集团);Institute of High Performance Computing, A*STAR(新加坡科技研究局高性能计算研究所)
Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School(马萨诸塞州总医院与哈佛医学院戈登医学影像中心);Korea Advanced Institute of Science and Technology(韩国科学技术院)
Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Clasification
Zhejiang University(浙江大学);JD Health International Inc.(京东健康国际有限公司);Beijing Huilongguan Hospital(北京回龙观医院);Peking University Huilongguan Clinical Medical School(北京大学回龙观临床医学院)
mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Tencent(腾讯);University of California, Los Angeles(加州大学洛杉矶分校);Lenovo Research(联想研究院)
A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers
Centre des Matériaux (MAT)(材料中心);CNRS(法国国家科学研究中心);Centre de Mise en Forme des Matériaux (CEMEF)(材料成型中心);Centre de Morphologie Mathématique (CMM)(数学形态学中心);Mines Paris(巴黎高等矿业学院);Université PSL(巴黎文理研究大学)
Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging
FocusNet: Transformer-enhanced Polyp Segmentation with Local and Pooling Attention
Chongqing University of Posts and Telecommunications(重庆邮电大学);University of South Dakota(南达科他大学);Malaviya National Institute of Technology(马拉维亚国立理工学院);Harvard Medical School(哈佛医学院);University of Gothenburg(戈尔堡大学)
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives
Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis
ShanghaiTech University(上海科技大学);Nanjing University of Science and Technology(南京理工大学);Nanjing University(南京大学);National Institute of Healthcare Data Science at Nanjing University(南京大学医疗健康数据科学国家研究院);Southeast University(东南大学);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能有限公司);Shanghai Clinical Research and Trial Center(上海临床研究中心)
AMGFormer: Adaptive Multi-Granular Transformer for Brain Tumor Segmentation with Missing Modalities
1 Faculty of Information Engineering;Automation, Kunming University of Science;Technology 2 Hepatobiliary \& Pancreatic Surgery, First People's Hospital of Yunnan/KUST Affiliated Hospital
Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data
Queensland Brain Institute, The University of Queensland(昆士兰脑研究所,昆士兰大学);School of Information Technology and Electrical Engineering, The University of Queensland(信息技术与电气工程学院,昆士兰大学)