Magnetic resonance imaging (MRI) is a widely used non-radiative and non-invasive method for clinical interrogation of organ structures and metabolism, with an inherently long scanning time. Methods by k-space undersampling and deep learning based reconstruction have been popularised to accelerate the scanning process. This work focuses on investigating how powerful transformers are for fast MRI by exploiting and comparing different novel network architectures. In particular, a generative adversarial network (GAN) based Swin transformer (ST-GAN) was introduced for the fast MRI reconstruction. To further preserve the edge and texture information, edge enhanced GAN based Swin transformer (EES-GAN) and texture enhanced GAN based Swin transformer (TES-GAN) were also developed, where a dual-discriminator GAN structure was applied. We compared our proposed GAN based transformers, standalone Swin transformer and other convolutional neural networks based GAN model in terms of the evaluation metrics PSNR, SSIM and FID. We showed that transformers work well for the MRI reconstruction from different undersampling conditions. The utilisation of GAN's adversarial structure improves the quality of images reconstructed when undersampled for 30% or higher. The code is publicly available at https://github.com/ayanglab/SwinGANMR.
Columbia University（哥伦比亚大学）；Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心）
Magnetic resonance imaging (MRI) noninvasively provides critical information about how human brain structures develop across stages of life. Developmental scientists are particularly interested in the first few years of neurodevelopment. Despite the success of MRI collection and analysis for adults, it is a challenge for researchers to collect high-quality multimodal MRIs from developing infants mainly because of their irregular sleep pattern, limited attention, inability to follow instructions to stay still, and a lack of analysis approaches. These challenges often lead to a significant reduc…
NVIDIA Corporation（英伟达公司）；Seoul National University（首尔大学）
Synthesizing medical images, such as PET, is a challenging task due to the fact that the intensity range is much wider and denser than those in photographs and digital renderings and are often heavily biased toward zero. Above all, intensity values in PET have absolute significance, and are used to compute parameters that are reproducible across the population. Yet, usually much manual adjustment has to be made in pre-/post- processing when synthesizing PET images, because its intensity ranges can vary a lot, e.g., between -100 to 1000 in floating point values. To overcome these challenges, we…
Neuroimaging of large populations is valuable to identify factors that promote or resist brain disease, and to assist diagnosis, subtyping, and prognosis. Data-driven models such as convolutional neural networks (CNNs) have increasingly been applied to brain images to perform diagnostic and prognostic tasks by learning robust features. Vision transformers (ViT) - a new class of deep learning architectures - have emerged in recent years as an alternative to CNNs for several computer vision applications. Here we tested variants of the ViT architecture for a range of desired neuroimaging downstre…
National Heart and Lung Institute, Imperial College London（帝国理工学院国家心肺研究所）；Imperial College London（帝国理工学院）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）；Brunel University London（伦敦布鲁内尔大学）；University of Cambridge（剑桥大学）；Technical University of Munich（慕尼黑工业大学）；Weizmann Institute of Science（魏茨曼科学研究所）
Research studies have shown no qualms about using data driven deep learning models for downstream tasks in medical image analysis, e.g., anatomy segmentation and lesion detection, disease diagnosis and prognosis, and treatment planning. However, deep learning models are not the sovereign remedy for medical image analysis when the upstream imaging is not being conducted properly (with artefacts). This has been manifested in MRI studies, where the scanning is typically slow, prone to motion artefacts, with a relatively low signal to noise ratio, and poor spatial and/or temporal resolution. Recen…
SHFormer: Dynamic Spectral Filtering Convolutional Neural Network and High-pass Kernel Generation Transformer for Adaptive MRI Reconstruction
Indian Institute of Technology Madras (IITM)（印度理工学院马德拉斯分校）；Healthcare Technology Innovation Centre（医疗技术创新中心）；GE John F Welch Technology Center（通用电气约翰·F·韦尔奇技术中心）
CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University（海德堡大学医学院心血管医学人工智能研究所，海德堡大学）；Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital（海德堡大学医院心脏病学、血管学、肺病学系）；DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim（德国心血管研究中心（DZHK），海德堡/曼海姆合作站点）；Department of Radiology and Nuclear Medicine, University Medical Center Mannheim（曼海姆大学医学中心放射学与核医学系）；Section for Invasive Cardiology and Electrophysiology, I. Medical Department, Cardiology, Hemostasiology and Intensive Care, University Medical Centre Mannheim（曼海姆大学医学中心第一内科（心脏病学、止血学与重症监护）介入心脏病学与电生理学科）；University Medical Center of the Johannes Gutenberg-University Mainz（美因茨约翰内斯·古腾堡大学医学中心）；Department of Cardiology, Angiology, Hemostasis, and Medical Intensive Care, University Medical Centre Mannheim, Medical Faculty Mannheim, University of Heidelberg（海德堡大学曼海姆医学院曼海姆大学医学中心心脏病学、血管学、止血学与内科重症监护系）
本文提出CardioDiT，一种基于扩散变换器的4D潜在扩散模型，用于合成短轴 cine CMR，通过联合建模空间和时间实现连续心脏动态生成。
Johns Hopkins University（约翰斯·霍普金斯大学）；Subtle Medical Inc.（Subtle医疗公司）；Stanford University（斯坦福大学）
Multi-contrast magnetic resonance imaging (MRI) is widely used in clinical practice as each contrast provides complementary information. However, the availability of each imaging contrast may vary amongst patients, which poses challenges to radiologists and automated image analysis algorithms. A general approach for tackling this problem is missing data imputation, which aims to synthesize the missing contrasts from existing ones. While several convolutional neural networks (CNN) based algorithms have been proposed, they suffer from the fundamental limitations of CNN models, such as the requir…
National Heart and Lung Institute, Imperial College London（帝国理工学院国家心肺研究所）；Cardiovascular Research Centre, Royal Brompton Hospital（皇家布朗普顿医院心血管研究中心）；School of Biomedical Engineering, Sun Yat-sen University（中山大学生物医学工程学院）
Fast MRI aims to reconstruct a high fidelity image from partially observed measurements. Exuberant development in fast MRI using deep learning has been witnessed recently. Meanwhile, novel deep learning paradigms, e.g., Transformer based models, are fast-growing in natural language processing and promptly developed for computer vision and medical image analysis due to their prominent performance. Nevertheless, due to the complexity of the Transformer, the application of fast MRI may not be straightforward. The main obstacle is the computational cost of the self-attention layer, which is the co…
Yale University（耶鲁大学）；New York University（纽约大学）；Hyperfine Research（Hyperfine研究公司）
Multi-contrast MRI (MC-MRI) captures multiple complementary imaging modalities to aid in radiological decision-making. Given the need for lowering the time cost of multiple acquisitions, current deep accelerated MRI reconstruction networks focus on exploiting the redundancy between multiple contrasts. However, existing works are largely supervised with paired data and/or prohibitively expensive fully-sampled MRI sequences. Further, reconstruction networks typically rely on convolutional architectures which are limited in their capacity to model long-range interactions and may lead to suboptima…
Bilkent University（比尔肯特大学）；National Magnetic Resonance Research Center, Bilkent University（比尔肯特大学国家磁共振研究中心）
Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here …
College of Computer Science and Technology（计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Generative Adversarial Networks (GANs) are powerful tools for reconstructing Compressed Sensing Magnetic Resonance Imaging (CS-MRI). However most recent works lack exploration of structure information of MRI images that is crucial for clinical diagnosis. To tackle this problem, we propose the Structure-Enhanced GAN (SEGAN) that aims at restoring structure information at both local and global scale. SEGAN defines a new structure regularization called Patch Correlation Regularization (PCR) which allows for efficient extraction of structure information. In addition, to further enhance the ability…
Monash Biomedical Imaging, Monash University（莫纳什大学生物医学成像中心）；Department of Electrical and Computer Systems Engineering, Monash University（莫纳什大学电气与计算机系统工程系）；School of Psychological Sciences, Monash University（莫纳什大学心理科学学院）；Department of Data Science and AI, Monash University（莫纳什大学数据科学与人工智能系）
Global correlations are widely seen in human anatomical structures due to similarity across tissues and bones. These correlations are reflected in magnetic resonance imaging (MRI) scans as a result of close-range proton density and T1/T2 parameters. Furthermore, to achieve accelerated MRI, k-space data are undersampled which causes global aliasing artifacts. Convolutional neural network (CNN) models are widely utilized for accelerated MRI reconstruction, but those models are limited in capturing global correlations due to the intrinsic locality of the convolution operation. The self-attention-…
Image-based brain cancer prediction models, based on radiomics, quantify the radiologic phenotype from magnetic resonance imaging (MRI). However, these features are difficult to reproduce because of variability in acquisition and preprocessing pipelines. Despite evidence of intra-tumor phenotypic heterogeneity, the spatial diversity between different slices within an MRI scan has been relatively unexplored using such methods. In this work, we propose a deep anchor attention aggregation strategy with a Vision Transformer to predict survival risk for brain cancer patients. A Deep Anchor Attentio…
University of Toronto（多伦多大学）；The Hospital for Sick Children（病童医院）
Many high-performance classification models utilize complex CNN-based architectures for Alzheimer's Disease classification. We aim to investigate two relevant questions regarding classification of Alzheimer's Disease using MRI: "Do Vision Transformer-based models perform better than CNN-based models?" and "Is it possible to use a shallow 3D CNN-based model to obtain satisfying results?" To achieve these goals, we propose two models that can take in and process 3D MRI scans: Convolutional Voxel Vision Transformer (CVVT) architecture, and ConvNet3D-4, a shallow 4-block 3D CNN-based model. Our re…
Accelerating magnetic resonance image (MRI) reconstruction process is a challenging ill-posed inverse problem due to the excessive under-sampling operation in k-space. In this paper, we propose a recurrent transformer model, namely ReconFormer, for MRI reconstruction which can iteratively reconstruct high fertility magnetic resonance images from highly under-sampled k-space data. In particular, the proposed architecture is built upon Recurrent Pyramid Transformer Layers (RPTL), which jointly exploits intrinsic multi-scale information at every architecture unit as well as the dependencies of th…
Harbin Institute of Technology（哈尔滨工业大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Nanjing Medical University（南京医科大学）
The core problem of Magnetic Resonance Imaging (MRI) is the trade off between acceleration and image quality. Image reconstruction and super-resolution are two crucial techniques in Magnetic Resonance Imaging (MRI). Current methods are designed to perform these tasks separately, ignoring the correlations between them. In this work, we propose an end-to-end task transformer network (T$^2$Net) for joint MRI reconstruction and super-resolution, which allows representations and feature transmission to be shared between multiple task to achieve higher-quality, super-resolved and motion-artifacts-fr…
This paper considers the problem of undersampled MRI reconstruction. We propose a novel Transformer-based framework for directly processing signal in k-space, going beyond the limitation of regular grids as ConvNets do. We adopt an implicit representation of k-space spectrogram, treating spatial coordinates as inputs, and dynamically query the sparsely sampled points to reconstruct the spectrogram, i.e. learning the inductive bias in k-space. To strike a balance between computational cost and reconstruction quality, we build the decoder with hierarchical structure to generate low-resolution an…
Otto-von-Guericke University（奥托·冯·格里克大学）；University Hospital of Magdeburg（马格德堡大学医院）
Deep learning-based segmentation of the liver and hepatic lesions therein steadily gains relevance in clinical practice due to the increasing incidence of liver cancer each year. Whereas various network variants with overall promising results in the field of medical image segmentation have been successfully developed over the last years, almost all of them struggle with the challenge of accurately segmenting hepatic lesions in magnetic resonance imaging (MRI). This led to the idea of combining elements of convolutional and transformer-based architectures to overcome the existing limitations. T…
Convolutional neural networks (CNNs) have achieved remarkable success in automatically segmenting organs or lesions on 3D medical images. Recently, vision transformer networks have exhibited exceptional performance in 2D image classification tasks. Compared with CNNs, transformer networks have an appealing advantage of extracting long-range features due to their self-attention algorithm. Therefore, we propose a CNN-Transformer combined model, called BiTr-Unet, with specific modifications for brain tumor segmentation on multi-modal MRI scans. Our BiTr-Unet achieves good performance on the BraTS…
Semantic segmentation of brain tumors is a fundamental medical image analysis task involving multiple MRI imaging modalities that can assist clinicians in diagnosing the patient and successively studying the progression of the malignant entity. In recent years, Fully Convolutional Neural Networks (FCNNs) approaches have become the de facto standard for 3D medical image segmentation. The popular "U-shaped" network architecture has achieved state-of-the-art performance benchmarks on different 2D and 3D semantic segmentation tasks and across various imaging modalities. However, due to the limited…
Residual Vision Transformer (ResViT) Based Self-Supervised Learning Model for Brain Tumor Classification
Center for Diabetes Technology（糖尿病技术中心）；University of Virginia（弗吉尼亚大学）；Erciyes University（埃尔齐耶斯大学）；Biocomplexity Institute and Initiative（生物复杂性研究所与倡议项目）
In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements. Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely on convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision task…
University of Surrey（萨里大学）；Children’s National Hospital（国家儿童医院）；Surrey Institute for People-Centred AI（萨里以人为本人工智能研究所）
The availability of large scale data with high quality ground truth labels is a challenge when developing supervised machine learning solutions for healthcare domain. Although, the amount of digital data in clinical workflows is increasing, most of this data is distributed on clinical sites and protected to ensure patient privacy. Radiological readings and dealing with large-scale clinical data puts a significant burden on the available resources, and this is where machine learning and artificial intelligence play a pivotal role. Magnetic Resonance Imaging (MRI) for musculoskeletal (MSK) diagn…
We apply an ensemble of modified TransBTS, nnU-Net, and a combination of both for the segmentation task of the BraTS 2021 challenge. In fact, we change the original architecture of the TransBTS model by adding Squeeze-and-Excitation blocks, an increasing number of CNN layers, replacing positional encoding in Transformer block with a learnable Multilayer Perceptron (MLP) embeddings, which makes Transformer adjustable to any input size during inference. With these modifications, we are able to largely improve TransBTS performance. Inspired by a nnU-Net framework we decided to combine it with our…
Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning
School of Artificial Intelligence, Hebei University of Technology（河北工业大学人工智能学院）；The First Hospital of Jilin University（吉林大学第一医院）；Research Center for Medical AI, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院医学人工智能研究中心）
Identification of lymph nodes (LN) in T2 Magnetic Resonance Imaging (MRI) is an important step performed by radiologists during the assessment of lymphoproliferative diseases. The size of the nodes play a crucial role in their staging, and radiologists sometimes use an additional contrast sequence such as diffusion weighted imaging (DWI) for confirmation. However, lymph nodes have diverse appearances in T2 MRI scans, making it tough to stage for metastasis. Furthermore, radiologists often miss smaller metastatic lymph nodes over the course of a busy day. To deal with these issues, we propose t…
Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data
Digital Technology and Innovation, Siemens Healthineers（西门子医疗数字技术与创新部）；Pattern Recognition Lab, Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（弗赖堡-艾尔朗根-纽伦堡大学计算机科学系模式识别实验室）
Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis
School of Engineering, University of Birmingham, UK（伯明翰大学工程学院）；William Harvey Research Institute, Queen Mary University London, UK（女王玛丽大学伦敦威廉·哈里维研究所）；Barts Heart Centre, St Bartholomew’s Hospital, Barts Health NHS Trust, UK（巴特勒心脏中心，圣巴塞洛缪医院，巴特勒健康 NHS信托）；Division of Cardiology, Johns Hopkins University School of Medicine, US（约翰霍普金斯大学医学院心脏病科）
Despite its tremendous value for the diagnosis, treatment monitoring and surveillance of children with cancer, whole body staging with positron emission tomography (PET) is time consuming and associated with considerable radiation exposure. 100x (1% of the standard clinical dosage) ultra-low-dose/ultra-fast whole-body PET reconstruction has the potential for cancer imaging with unprecedented speed and improved safety, but it cannot be achieved by the naive use of machine learning techniques. In this study, we utilize the global similarity between baseline and follow-up PET and magnetic resonan…
University of California, Los Angeles (UCLA)（加利福尼亚大学洛杉矶分校）；The First Affiliated Hospital of China Medical University（中国医科大学附属第一医院）；VoxelCloud, Inc.（体素云公司）
Prostate cancer is the second leading cause of cancer death among men in the United States. The diagnosis of prostate MRI often relies on the accurate prostate zonal segmentation. However, state-of-the-art automatic segmentation methods often fail to produce well-contained volumetric segmentation of the prostate zones since certain slices of prostate MRI, such as base and apex slices, are harder to segment than other slices. This difficulty can be overcome by accounting for the cross-slice relationship of adjacent slices, but current methods do not fully learn and exploit such relationships. I…
MIT（麻省理工学院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；University College London（伦敦大学学院）；Massachusetts General Hospital（麻省总医院）
Volumetric reconstruction of fetal brains from multiple stacks of MR slices, acquired in the presence of almost unpredictable and often severe subject motion, is a challenging task that is highly sensitive to the initialization of slice-to-volume transformations. We propose a novel slice-to-volume registration method using Transformers trained on synthetically transformed data, which model multiple stacks of MR slices as a sequence. With the attention mechanism, our model automatically detects the relevance between slices and predicts the transformation of one slice using information from othe…
Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Sun Yat-sen University（中山大学）；Beihang University（北京航空航天大学）；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））；TECNALIA（泰克纳利亚研究中心）；Shenzhen Second People’s Hospital（深圳市第二人民医院）；The First Affiliated Hospital of Shenzhen University Health Science Center（深圳大学医学部第一附属医院）
Magnetic resonance imaging (MRI) is an important non-invasive clinical tool that can produce high-resolution and reproducible images. However, a long scanning time is required for high-quality MR images, which leads to exhaustion and discomfort of patients, inducing more artefacts due to voluntary movements of the patients and involuntary physiological movements. To accelerate the scanning process, methods by k-space undersampling and deep learning based reconstruction have been popularised. This work introduced SwinMR, a novel Swin transformer based method for fast MRI reconstruction. The who…
