Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.
SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging
Deepwise Healthcare（深睿医疗）；Sir Run Run Shaw Hospital, Zhejiang University School of Medicine（浙江大学医学院附属邵逸夫医院）；Ningbo Medical Center Lihuili Hospital（宁波医疗中心李惠利医院）；The University of Hong Kong（香港大学）
VT-Former: Efffcient Transformer-based Decoder for Varshamov-Tenengolts Codes
Center for Applied Mathematics, KL-AAGDM, Tianjin University（天津大学应用数学中心、KL-AAGDM）；State Key Laboratory of Synthetic Biology, Tianjin University（天津大学合成生物学国家重点实验室）
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
Concordia University（康考迪亚大学）；Khalifa University（哈利法大学）；Princess Sumaya University for Technology（苏玛雅公主科技大学）；University of Alberta（阿尔伯塔大学）；Polish Academy of Sciences（波兰科学院）；Istinye University（伊斯廷耶大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）
NN-Former: Rethinking Graph Structure in Neural Architecture Representation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；Northwestern Polytechnical University（西北工业大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
School of Software, Dalian University of Technology（大连理工大学软件学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）；Institute of Innovation, Science and Sustainability, Federation University Australia（联邦大学澳大利亚创新、科学与可持续性研究所）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；School of Foreign Languages and School of Software Technology, Dalian University of Technology（大连理工大学外语学院和软件技术学院）
A Survey of Graph Transformers: Architectures, Theories and Applications
The Chinese University of Hong Kong（香港中文大学）；SIGS, Tsinghua University（清华大学深圳国际研究生院）；ISTI-CNR（意大利国家研究委员会信息科学与技术研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Hupan Lab（湖畔实验室）；Renmin University of China（中国人民大学）
Causal-Transformer with Adaptive Mutation-Locking for Early Prediction of Acute Kidney Injury
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Department of Nephrology, TEDA Hospital, Tianjin University（天津大学TEDA医院肾内科）；Tianjin Medical University General Hospital（天津医科大学总医院）；Department of Nephrology, First Teaching Hospital of Tianjin University of Traditional Chinese Medicine（天津中医药大学第一教学医院肾内科）
From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
United Arab Emirates University（阿联酋大学）；Emirates Center for Mobility Research（阿联酋移动性研究中心）
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computing and Mathematical Sciences, Birkbeck, University of London（伦敦大学伯克贝克学院计算与数学科学学院）
MBZ University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Monash University（莫纳什大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；University of Central Florida（中佛罗里达大学）
Department of Electronics and Computer Engineering（电子与计算机工程系）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）；School of Computer Science and Technology（计算机科学与技术学院）
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；University of Michigan（密歇根大学）；Xiamen University（厦门大学）；The Ohio State University（俄亥俄州立大学）；Columbia University（哥伦比亚大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）
Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Survey
Renmin University of China（中国人民大学）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）；Department of houmo.ai（houmo.ai部门）
A State-of-the-art Survey of Artificial Neural Networks for Whole-slide Image Analysis:from Popular Convolutional Neural Networks to Potential Visual Transformers
Northeastern University（东北大学）；Chengdu University of Information Technology（成都信息工程大学）；China Medical University（中国医科大学）；University of Luebeck（吕贝克大学）
