Multi-sequence MRIs can be necessary for reliable diagnosis in clinical practice due to the complimentary information within sequences. However, redundant information exists across sequences, which interferes with mining efficient representations by modern machine learning or deep learning models. To handle various clinical scenarios, we propose a sequence-to-sequence generation framework (Seq2Seq) for imaging-differentiation representation learning. In this study, not only do we propose arbitrary 3D/4D sequence generation within one model to generate any specified target sequence, but also we are able to rank the importance of each sequence based on a new metric estimating the difficulty of a sequence being generated. Furthermore, we also exploit the generation inability of the model to extract regions that contain unique information for each sequence. We conduct extensive experiments using three datasets including a toy dataset of 20,000 simulated subjects, a brain MRI dataset of 1,251 subjects, and a breast MRI dataset of 2,101 subjects, to demonstrate that (1) our proposed Seq2Seq is efficient and lightweight for complex clinical datasets and can achieve excellent image quality; (2) top-ranking sequences can be used to replace complete sequences with non-inferior performance; (3) combining MRI with our imaging-differentiation map leads to better performance in clinical tasks such as glioblastoma MGMT promoter methylation status prediction and breast cancer pathological complete response status prediction. Our code is available at https://github.com/fiy2W/mri_seq2seq.
Eastern Hepatobiliary Surgery Hospital（东方肝胆外科医院）；Ping An Technology（平安科技）；Georgia Institute of Technology（佐治亚理工学院）；PAII Inc.（PAII公司）
Objectives: to propose a fully-automatic computer-aided diagnosis (CAD) solution for liver lesion characterization, with uncertainty estimation. Methods: we enrolled 400 patients who had either liver resection or a biopsy and was diagnosed with either hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma, or secondary metastasis, from 2006 to 2019. Each patient was scanned with T1WI, T2WI, T1WI venous phase (T2WI-V), T1WI arterial phase (T1WI-A), and DWI MRI sequences. We propose a fully-automatic deep CAD pipeline that localizes lesions from 3D MRI studies using key-slice parsing an…
Exploiting Completeness Perception with Diffusion Transformer for Unified 3D MRI Synthesis
School of Engineering, University of Birmingham, UK（伯明翰大学工程学院）；William Harvey Research Institute, Queen Mary University London, UK（女王玛丽大学伦敦威廉·哈里维研究所）；Barts Heart Centre, St Bartholomew’s Hospital, Barts Health NHS Trust, UK（巴特勒心脏中心，圣巴塞洛缪医院，巴特勒健康 NHS信托）；Division of Cardiology, Johns Hopkins University School of Medicine, US（约翰霍普金斯大学医学院心脏病科）
An Explainable Deep Framework: Towards Task-Specific Fusion for Multi-to-One MRI Synthesis
Radboud University Medical Centre（拉德堡德大学医学中心）；Netherlands Cancer Institute (NKI)（荷兰癌症研究所）；Maastricht University（马斯特里赫特大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；University of Leeds（利兹大学）
Bilkent University（比尔肯特大学）；University of Southern California（南加州大学）
Multi-institutional collaborations are key for learning generalizable MRI synthesis models that translate source- onto target-contrast images. To facilitate collaboration, federated learning (FL) adopts decentralized training and mitigates privacy concerns by avoiding sharing of imaging data. However, FL-trained synthesis models can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident when common or variable translation tasks are prescribed across sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) to improve reliabil…
Pre- to Post-Contrast Breast MRI Synthesis for Enhanced Tumour Segmentation
Universitat de Barcelona（巴塞罗那大学）；Helmholtz Center Munich（慕尼黑亥姆霍兹中心）；Technical University of Munich（慕尼黑工业大学）；Karolinska Institutet（卡罗林斯卡学院）；King’s College London（伦敦国王学院）
Bilkent University（比尔肯特大学）；University of Southern California（南加利福尼亚大学）；Hacettepe University（哈杰泰佩大学）
Learning-based synthetic multi-contrast MRI commonly involves deep models trained using high-quality images of source and target contrasts, regardless of whether source and target domain samples are paired or unpaired. This results in undesirable reliance on fully-sampled acquisitions of all MRI contrasts, which might prove impractical due to limitations on scan costs and time. Here, we propose a novel semi-supervised deep generative model that instead learns to recover high-quality target images directly from accelerated acquisitions of source and target contrasts. To achieve this, the propos…
Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning
School of Artificial Intelligence, Hebei University of Technology（河北工业大学人工智能学院）；The First Hospital of Jilin University（吉林大学第一医院）；Research Center for Medical AI, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院医学人工智能研究中心）
University of Florida（佛罗里达大学）；Georgia State University（佐治亚州立大学）
Generating multi-contrasts/modal MRI of the same anatomy enriches diagnostic information but is limited in practice due to excessive data acquisition time. In this paper, we propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI using incomplete k-space data of several source modalities as inputs. The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality. Our proposed model is formulated as a variational problem that leverages several learnable modality-specific feature extract…
CAVM: Conditional Autoregressive Vision Model for Contrast-Enhanced Brain Tumor MRI Synthesis
School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）
Mamba-driven MRI-to-CT Synthesis for MRI-only Radiotherapy Planning
School of Electrical；Computer Engineering, National Technical University of Athens Archimedes, Athena Research Center CentraleSup\'elec, University of Paris-Saclay
Fetal brain magnetic resonance imaging (MRI) offers exquisite images of the developing brain but is not suitable for anomaly screening. For this ultrasound (US) is employed. While expert sonographers are adept at reading US images, MR images are much easier for non-experts to interpret. Hence in this paper we seek to produce images with MRI-like appearance directly from clinical US images. Our own clinical motivation is to seek a way to communicate US findings to patients or clinical professionals unfamiliar with US, but in medical image analysis such a capability is potentially useful, for in…
Synthesis of Contrast-Enhanced Breast MRI Using Multi-b-Value DWI-based Hierarchical Fusion Network with Attention Mechanism
Netherlands Cancer Institute (NKI)（荷兰癌症研究所）；Maastricht University（马斯特里赫特大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Fondazione Universitaria A. Gemelli, IRCCS Roma（罗马A.杰梅利大学基金会IRCCS）
SAFE-Diff: Scale-Aware Attention and Feature-Dispersive Diffusion with Uncertainty Estimation for Contrast-Enhanced Breast MRI Synthesis
Department of Medical Imaging, Radboud University Medical Center（鲁文大学医学中心医学影像部）；Department of Radiology, Netherlands Cancer Institute（荷兰癌症研究所放射科）；Maastro Clinic（马斯垂克诊所）；Faculty of Applied Science, Macao Polytechnic University（澳门理工大学应用科学学院）；Department of Radiation Oncology, Netherlands Cancer Institute（荷兰癌症研究所放射肿瘤科）
Current deep learning approaches for diffusion MRI modeling circumvent the need for densely-sampled diffusion-weighted images (DWIs) by directly predicting microstructural indices from sparsely-sampled DWIs. However, they implicitly make unrealistic assumptions of static $q$-space sampling during training and reconstruction. Further, such approaches can restrict downstream usage of variably sampled DWIs for usages including the estimation of microstructural indices or tractography. We propose a generative adversarial translation framework for high-quality DWI synthesis with arbitrary $q$-space…
Recent studies suggest that combined analysis of Magnetic resonance imaging~(MRI) that measures brain atrophy and positron emission tomography~(PET) that quantifies hypo-metabolism provides improved accuracy in diagnosing Alzheimer's disease. However, such techniques are limited by the availability of corresponding scans of each modality. Current work focuses on a cross-modal approach to estimate FDG-PET scans for the given MR scans using a 3D U-Net architecture. The use of the complete MR image instead of a local patch based approach helps in capturing non-local and non-linear correlations be…
A Proof-of-Concept Study of Multitask Learning for Cranial Synthetic CT Generation Across Heterogeneous MRI Field Strengths
F.M. Kirby Research Center for Brain Imaging, Kennedy Krieger Institute（F.M. Kirby脑成像研究中心，Kennedy Krieger研究所）；Neurosection, Division of MR Research, Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine（神经科，磁共振研究部，约翰·霍普金斯大学医学院放射学与放射科学系）；Department of Biomedical Engineering, Johns Hopkins University（生物医学工程系，约翰·霍普金斯大学）；Department of Biomedical Engineering, Case Western Reserve University（生物医学工程系，凯斯西储大学）；Department of Biomedical Engineering, Columbia University（生物医学工程系，哥伦比亚大学）；Department of Neuroscience, Columbia University（神经科学系，哥伦比亚大学）；Department of Radiology, Mayo Clinic（放射科，梅奥诊所）；Neuroradiology and Neurosurgery, Mayo Clinic College of Medicine and Science（神经放射学与神经外科，梅奥诊所医学院与科学学院）
Stanford University（斯坦福大学）；GE Healthcare（通用电气医疗集团）；University of California Riverside（加州大学河滨分校）
Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of a wide range of neurological diseases. Positron emission tomography (PET) with radiolabeled water (15O-water) is considered the gold-standard for the measurement of CBF in humans. PET imaging, however, is not widely available because of its prohibitive costs and use of short-lived radiopharmaceutical tracers that typically require onsite cyclotron production. Magnetic resonance imaging (MRI), in contrast, is more readily accessible and does not involve ionizing radiation. This study presents a…
When Brains Disagree: Biological Ambiguity Underlies the Challenge of Amyloid PET Synthesis from Structural MRI
Hawkes Institute, University College London, UK（霍克斯研究所，伦敦大学学院，英国）；Department of Medical Physics and Biomedical Engineering, University College London, UK（医学物理与生物医学工程系，伦敦大学学院，英国）；AINOSTICS Ltd, Manchester, UK（AINOSTICS有限公司，曼彻斯特，英国）；Dementia Research Centre, UCL Queen Square Institute of Neurology, University College London, UK（痴呆研究中心，伦敦大学学院女王广场神经科学研究所，英国）；UK Dementia Research Institute, London, UK（英国痴呆研究研究所，伦敦，英国）；Department of Computer Science, University College London, UK（计算机科学系，伦敦大学学院，英国）
Accurate synthesis of a full 3D MR image containing tumours from available MRI (e.g. to replace an image that is currently unavailable or corrupted) would provide a clinician as well as downstream inference methods with important complementary information for disease analysis. In this paper, we present an end-to-end 3D convolution neural network that takes a set of acquired MR image sequences (e.g. T1, T2, T1ce) as input and concurrently performs (1) regression of the missing full resolution 3D MRI (e.g. FLAIR) and (2) segmentation of the tumour into subtypes (e.g. enhancement, core). The hypo…
Towards Personalized Multi-Modal MRI Synthesis across Heterogeneous Datasets
University of Science and Technology of China（中国科学技术大学）；University of Electronic Science and Technology of China（电子科技大学）；Capital Medical University（首都医科大学）；Army Medical University（中国人民解放军陆军军医大学）；China National Clinical Research Center for Neurological Diseases（中国国家神经疾病临床研究中心）
Cyclic Self-Supervised Diffusion for Ultra Low-field to High-field MRI Synthesis
Department of Bioengineering and Imperial-X, Imperial College London（生物工程系和Imperial-X，帝国理工学院伦敦分校）；Department of Bioengineering, Imperial College London（生物工程系，帝国理工学院伦敦分校）；School of Biomedical Engineering, Sun Yat-sen University（生物医学工程学院，中山大学）；Monash Biomedical Imaging, Monash University（莫纳什大学生物医学成像）
University College London（伦敦大学学院）；Biomediq A/S（Biomediq公司）；University of Copenhagen（哥本哈根大学）
Segmenting vascular pathologies such as white matter lesions in Brain magnetic resonance images (MRIs) require acquisition of multiple sequences such as T1-weighted (T1-w) --on which lesions appear hypointense-- and fluid attenuated inversion recovery (FLAIR) sequence --where lesions appear hyperintense--. However, most of the existing retrospective datasets do not consist of FLAIR sequences. Existing missing modality imputation methods separate the process of imputation, and the process of segmentation. In this paper, we propose a method to link both modality imputation and segmentation using…
University College London（伦敦大学学院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Allen Institute for Brain Science（艾伦脑科学研究所）；Massachusetts Institute of Technology（麻省理工学院）
Nonlinear registration of 2D histological sections with corresponding slices of MRI data is a critical step of 3D histology reconstruction. This task is difficult due to the large differences in image contrast and resolution, as well as the complex nonrigid distortions produced when sectioning the sample and mounting it on the glass slide. It has been shown in brain MRI registration that better spatial alignment across modalities can be obtained by synthesizing one modality from the other and then using intra-modality registration metrics, rather than by using mutual information (MI) as metric…
Large-Scale Pre-training Enables Multimodal AI Differentiation of Radiation Necrosis from Brain Metastasis Progression on Routine MRI
University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根大学医院，埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN)（埃尔朗根-欧洲大都会区纽伦堡综合癌症中心）；Bavarian Cancer Research Center (BZKF)（巴伐利亚癌症研究中心）；Universität Würzburg（维尔茨堡大学）；Peking University（北京大学）；Universitätsklinikum Würzburg（维尔茨堡大学医院）
AA-ViT: Anatomically Aware Vision Transformer with Structural and Frequency Guidance for Contrast Enhanced Brain MRI Synthesis
Atlantic Technological University（大西洋理工大学）；Belfast Health and Social Care Trust（贝尔法斯特健康和社会关怀信托基金）；King’s College Hospital NHS Foundation Trust（国王学院医院国民保健服务信托基金）；King’s College London（伦敦国王学院）；Queen’s University Belfast（贝尔法斯特女王大学）
Triple-Phase Sequential Fusion Network for Hepatobiliary Phase Liver MRI Synthesis
Yu-Yue Pathology Research Center, Jinfeng Laboratory（于越病理研究中心，锦峰实验室）；Department of Clinical Laboratory Medicine, The First Affiliated Hospital of Army Medical University（临床实验室医学部，陆军军医大学第一附属医院）；Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China（智能协同计算实验室，电子科技大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai Jiao Tong University（上海交通大学）
Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and risk, the acquisition of certain image modalities could be limited. To address this issue, many cross-modality medical image synthesis methods have been proposed. However, the current methods cannot well model the hard-to-synthesis regions (e.g., tumor or lesion regions). To address this issue, we propose a simple but effective strategy, that is, we propose a dual-discriminator (dual-D) adversarial learning system, in which, a global-D is used to make an overall eval…
FedSynthCT-Brain: A Federated Learning Framework for Multi-Institutional Brain MRI-to-CT Synthesis
Institute of Biomedical Engineering（生物医学工程研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工大学）；Department of Radiation Oncology（放射肿瘤科）；University Medical Center Schleswig-Holstein（什勒斯威-霍斯特大学医院）；Department of Experimental and Clinical Medicine（实验与临床医学系）；Magna Graecia University（马格拉西亚大学）；Department of Information Engineering（信息工程系）；Università Politecnica delle Marche（帕尔米拉理工大学）
University of Beira Interior（贝拉内格罗大学）；University of Missouri-Columbia（密苏里大学哥伦比亚分校）
Multiphase active contour based models are useful in identifying multiple regions with different characteristics such as the mean values of regions. This is relevant in brain magnetic resonance images (MRIs), allowing the differentiation of white matter against gray matter. We consider a well defined globally convex formulation of Vese and Chan multiphase active contour model for segmenting brain MRI images. A well-established theory and an efficient dual minimization scheme are thoroughly described which guarantees optimal solutions and provides stable segmentations. Moreover, under the dual …
MRI-to-CT Synthesis With Cranial Suture Segmentations Using A Variational Autoencoder Framework
Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC, USA（Sheikh Zayed儿童外科创新研究所，Children’s National医院，华盛顿特区，美国）；CIBORG Lab, Department of Radiology, Children’s Hospital Los Angeles, Los Angeles, CA, USA（CIBORG实验室，放射科，Children’s Hospital Los Angeles，洛杉矶，加利福尼亚州，美国）；Departments of Radiology and Biomedical Engineering, University of Southern California, Los Angeles, CA, USA（放射科和生物医学工程系，美国南加州大学，洛杉矶，加利福尼亚州，美国）；School of Medicine and Health Sciences, George Washington University, Washington, DC, USA（医学与健康科学学院，乔治华盛顿大学，华盛顿特区，美国）
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland School of Dentistry（马里兰大学牙医学院）
Cycle reconstruction regularized adversarial training -- e.g., CycleGAN, DiscoGAN, and DualGAN -- has been widely used for image style transfer with unpaired training data. Several recent works, however, have shown that local distortions are frequent, and structural consistency cannot be guaranteed. Targeting this issue, prior works usually relied on additional segmentation or consistent feature extraction steps that are task-specific. To counter this, this work aims to learn a general add-on structural feature extractor, by explicitly enforcing the structural alignment between an input and it…
Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine（首尔大学医学院癌症生物学跨学科项目）；Department of Radiology, Seoul National University Hospital（首尔大学医院放射科）；Department of Radiology, Seoul National University College of Medicine（首尔大学医学院放射科）；Healthcare AI Research Institute, Seoul National University Hospital（首尔大学医院医疗人工智能研究所）
Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；California Institute of Technology（加州理工学院）；Massachusetts Institute of Technology（麻省理工学院）
In neuroimaging, MRI tissue properties characterize underlying neurobiology, provide quantitative biomarkers for neurological disease detection and analysis, and can be used to synthesize arbitrary MRI contrasts. Estimating tissue properties from a single scan session using a protocol available on all clinical scanners promises to reduce scan time and cost, enable quantitative analysis in routine clinical scans and provide scan-independent biomarkers of disease. However, existing tissue properties estimation methods - most often $\mathbf{T_1}$ relaxation, $\mathbf{T_2^*}$ relaxation, and proto…
Research Institute of Computer Vision and Robotics, University of Girona（赫罗纳大学计算机视觉与机器人研究所）；Assiut University（艾斯尤特大学）；Vall d’Hebron University Hospital（瓦尔德希伯伦大学医院）
In this paper, we propose generating synthetic multiple sclerosis (MS) lesions on MRI images with the final aim to improve the performance of supervised machine learning algorithms, therefore avoiding the problem of the lack of available ground truth. We propose a two-input two-output fully convolutional neural network model for MS lesion synthesis in MRI images. The lesion information is encoded as discrete binary intensity level masks passed to the model and stacked with the input images. The model is trained end-to-end without the need for manually annotating the lesions in the training set…
École de Technologie Supérieure（高等技术学院）；CHU Sainte-Justine Research Center（圣茹斯特医院研究中心）；Polytechnique Montréal（蒙特利尔理工学院）
Vertebral bone segmentation from magnetic resonance (MR) images is a challenging task. Due to the inherent nature of the modality to emphasize soft tissues of the body, common thresholding algorithms are ineffective in detecting bones in MR images. On the other hand, it is relatively easier to segment bones from CT images because of the high contrast between bones and the surrounding regions. For this reason, we perform a cross-modality synthesis between MR and CT domains for simple thresholding-based segmentation of the vertebral bones. However, this implicitly assumes the availability of pai…
TumorFlow: Physics-Guided Longitudinal MRI Synthesis of Glioblastoma Growth
Technical University of Munich (TUM), Germany（慕尼黑技术大学（TUM））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；AI for Image-Guided Diagnosis and Therapy（图像引导诊断与治疗的AI）；Chair for AI in Healthcare and Medicine, TUM and TUM University Hospital, Munich, Germany（医疗与医学中的AI主任，慕尼黑技术大学及慕尼黑大学医院）
Language-Enhanced Generative Modeling for Amyloid PET Synthesis from MRI and Blood Biomarkers
organization= Shanghai Artificial Intelligence Laboratory , city= Shanghai , postcode= 200082 , country= China；organization= School of Medicine, Xiamen University , city= Xiamen , state= Fujian , country= China；organization= Department of Nuclear Medicine \& PET Center, Huashan Hospital, Fudan University , city= Shanghai , country= China；organization= Department of Gerontology, Shanghai Jiao Tong University Affiliated Sixth People’s Hospital , city= Shanghai , country= China；organization= Department of Computer Science, Rutgers University , city= New Brunswick , state= New Jersey , country= United States；organization= Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences , city= Shenzhen , country= China
Huazhong University of Science and Technology（华中科技大学）；Hong Kong University of Science and Technology（香港科技大学）
In this paper, we address the problem of synthesizing multi-parameter magnetic resonance imaging (mp-MRI) data, i.e. Apparent Diffusion Coefficients (ADC) and T2-weighted (T2w), containing clinically significant (CS) prostate cancer (PCa) via semi-supervised adversarial learning. Specifically, our synthesizer generates mp-MRI data in a sequential manner: first generating ADC maps from 128-d latent vectors, followed by translating them to the T2w images. The synthesizer is trained in a semisupervised manner. In the supervised training process, a limited amount of paired ADC-T2w images and the c…
