Deep learning has become a prominent computational modeling tool in the areas of computer vision and image processing in recent years. This research comprehensively analyzes the different deep-learning methods used for image-to-image translation and reconstruction in the natural and medical imaging domains. We examine the famous deep learning frameworks, such as convolutional neural networks and generative adversarial networks, and their variants, delving into the fundamental principles and difficulties of each. In the field of natural computer vision, we investigate the development and extension of various deep-learning generative models. In comparison, we investigate the possible applications of deep learning to generative medical imaging problems, including medical image translation, MRI reconstruction, and multi-contrast MRI synthesis. This thorough review provides scholars and practitioners in the areas of generative computer vision and medical imaging with useful insights for summarizing past works and getting insight into future research paths.
Tel Aviv University（特拉维夫大学）；GE Research（GE研究院）；GE Healthcare（GE医疗）
Recent accelerated MRI reconstruction models have used Deep Neural Networks (DNNs) to reconstruct relatively high-quality images from highly undersampled k-space data, enabling much faster MRI scanning. However, these techniques sometimes struggle to reconstruct sharp images that preserve fine detail while maintaining a natural appearance. In this work, we enhance the image quality by using a Conditional Wasserstein Generative Adversarial Network combined with a novel Adaptive Gradient Balancing (AGB) technique that automates the process of combining the adversarial and pixel-wise terms and st…
We present a detailed description and reference implementation of preprocessing steps necessary to prepare the public Retrospective Image Registration Evaluation (RIRE) dataset for the task of magnetic resonance imaging (MRI) to X-ray computed tomography (CT) translation. Furthermore we describe and implement three state of the art convolutional neural network (CNN) and generative adversarial network (GAN) models where we report statistics and visual results of two of them.
Comparative Analysis of GAN and Diffusion for MRI-to-CT translation
Department of Computer Science, University of Copenhagen, Copenhagen, Denmark（哥本哈根大学计算机科学系）；Department of Oncology, Rigshospitalet, Copenhagen, Denmark（哥本哈根医院肿瘤科）
VinAI Research（VinAI研究院）；FPT University（FPT大学）；University of Arkansas（阿肯色大学）
Image synthesis from corrupted contrasts increases the diversity of diagnostic information available for many neurological diseases. Recently the image-to-image translation has experienced significant levels of interest within medical research, beginning with the successful use of the Generative Adversarial Network (GAN) to the introduction of cyclic constraint extended to multiple domains. However, in current approaches, there is no guarantee that the mapping between the two image domains would be unique or one-to-one. In this paper, we introduce a novel approach to unpaired image-to-image tr…
Influence of High-Performance Image-to-Image Translation Networks on Clinical Visual Assessment and Outcome Prediction: Utilizing Ultrasound to MRI Translation in Prostate Cancer
Multiscale Structure-Guided Latent Diffusion for Multimodal MRI Translation
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，沈阳，中国）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，沈阳，中国）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，沈阳，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学阿米人工智能研究所，埃德蒙顿，阿尔伯塔，加拿大）；AiShiWeiLai AI Research, Beijing, China（人工智能研究（北京）有限公司，北京，中国）
Reconstruct or Generate: Exploring the Spectrum of Generative Modeling for Cardiac MRI
School of Computation, Information and Technology（计算信息学院）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；School of Medicine, Klinikum Rechts der Isar（医学院，莱茵河右岸医院）；Department of Quantitative Biomedicine（定量生物医学系）；University of Zurich（苏黎世大学）；Department of Computing（计算系）；Imperial College London（伦敦帝国理工学院）
FS-RWKV: Leveraging Frequency Spatial-Aware RWKV for 3T-to-7T MRI Translation
Faculty of Science and Technology, University of Macau（澳门大学科学与技术学院）；School of Electronic and Communication Engineering, Shenzhen Polytechnic University（深圳职业技术学院电子与通信工程学院）；Department of Cardiology, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China（广东省人民医院心内科（广东省医学科学院））；Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China（广东省心血管病研究院，广东省人民医院，广东省医学科学院，广州，中国）
Plasma-CycleGAN: Plasma Biomarker-Guided MRI to PET Cross-modality Translation Using Conditional CycleGAN
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）；Basis Scottsdale（斯科茨代尔贝斯机构）；College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；Mayo Clinic Arizona（梅奥诊所亚利桑那州分院）
Research Institute of Beihang University in Shenzhen（北京航空航天大学深圳研究院）；Beihang University（北京航空航天大学）；Chongqing University（重庆大学）；Microsoft Research（微软研究院）
We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images without real acquisition. Our proposed method performs NeuroImage-to-NeuroImage translation (abbreviated as N2N) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity…
Inter-scanner and inter-protocol discrepancy in MRI datasets are known to lead to significant quantification variability. Hence image-to-image or scanner-to-scanner translation is a crucial frontier in the area of medical image analysis with a lot of potential applications. Nonetheless, a significant percentage of existing algorithms cannot explicitly exploit and preserve texture details from target scanners and offers individual solutions towards specialized task-specific architectures. In this paper, we design a multi-scale texture transfer to enrich the reconstruction images with more detai…
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（右伊萨尔医院）；University of Zurich（苏黎世大学）；Institut für KI und Informatik in der Medizin（医学人工智能与信息学研究所）；Imperial College London（伦敦帝国学院）
Recently, contrastive learning-based image translation methods have been proposed, which contrasts different spatial locations to enhance the spatial correspondence. However, the methods often ignore the diverse semantic relation within the images. To address this, here we propose a novel semantic relation consistency (SRC) regularization along with the decoupled contrastive learning, which utilize the diverse semantics by focusing on the heterogeneous semantics between the image patches of a single image. To further improve the performance, we present a hard negative mining by exploiting the …
Lesion-Aware Post-Training of Latent Diffusion Models for Synthesizing Diffusion MRI from CT Perfusion
College of Medicine, Seoul National University, Seoul, Republic of Korea（首尔国立大学医学院）；Department of Radiology, Seoul National University Hospital（首尔国立大学医院放射科）；Department of Biomedical Sciences, Seoul National University（首尔国立大学生物医学科学系）
University of Oxford（牛津大学）；Plexalis Ltd（Plexalis有限公司）
We propose a novel convolutional method for the detection and identification of vertebrae in whole spine MRIs. This involves using a learnt vector field to group detected vertebrae corners together into individual vertebral bodies and convolutional image-to-image translation followed by beam search to label vertebral levels in a self-consistent manner. The method can be applied without modification to lumbar, cervical and thoracic-only scans across a range of different MR sequences. The resulting system achieves 98.1% detection rate and 96.5% identification rate on a challenging clinical datas…
Universitat de Barcelona（巴塞罗那大学）；Barts Health NHS Trust（巴特兹健康国民保健服务信托基金）；Queen Mary University of London（伦敦玛丽女王大学）；Universitat Pompeu Fabra（庞培法布拉大学）；ICREA（加泰罗尼亚研究与高级研究院）
Accurate segmentation of the cardiac boundaries in late gadolinium enhancement magnetic resonance images (LGE-MRI) is a fundamental step for accurate quantification of scar tissue. However, while there are many solutions for automatic cardiac segmentation of cine images, the presence of scar tissue can make the correct delineation of the myocardium in LGE-MRI challenging even for human experts. As part of the Multi-Sequence Cardiac MR Segmentation Challenge, we propose a solution for LGE-MRI segmentation based on two components. First, a generative adversarial network is trained for the task o…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Image-to-image translation plays a vital role in tackling various medical imaging tasks such as attenuation correction, motion correction, undersampled reconstruction, and denoising. Generative adversarial networks have been shown to achieve the state-of-the-art in generating high fidelity images for these tasks. However, the state-of-the-art GAN-based frameworks do not estimate the uncertainty in the predictions made by the network that is essential for making informed medical decisions and subsequent revision by medical experts and has recently been shown to improve the performance and inter…
Cycle-guided Denoising Diffusion Probability Model for 3D Cross-modality MRI Synthesis
Emory University（埃默里大学）；Winship Cancer Institute（温希普癌症研究所）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Friedrich-Alexander Universität Erlangen-Nürnberg（弗里德里希-亚历山大大学埃尔兰根-纽伦堡分校）；Technical University of Applied Sciences Amberg-Weiden（安贝格-魏登应用技术大学）；Siemens Healthineers（西门子医疗）
Purpose A Magnetic Resonance Imaging (MRI) exam typically consists of several sequences that yield different image contrasts. Each sequence is parameterized through multiple acquisition parameters that influence image contrast, signal-to-noise ratio, acquisition time, and/or resolution. Depending on the clinical indication, different contrasts are required by the radiologist to make a diagnosis. As MR sequence acquisition is time consuming and acquired images may be corrupted due to motion, a method to synthesize MR images with adjustable contrast properties is required. Methods Therefore, we …
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；University of Rochester（罗切斯特大学）
Image-to-image translation tasks have been widely investigated with Generative Adversarial Networks (GANs). However, existing approaches are mostly designed in an unsupervised manner while little attention has been paid to domain information within unpaired data. In this paper, we treat domain information as explicit supervision and design an unpaired image-to-image translation framework, Domain-supervised GAN (DosGAN), which takes the first step towards the exploration of explicit domain supervision. In contrast to representing domain characteristics using different generators or domain codes…
Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization
University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术学院）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）
Self-Consistent Nested Diffusion Bridge for Accelerated MRI Reconstruction
School of Information Science and Technology, Fudan University, Shanghai, China（复旦大学信息科学与技术学院）；Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, Australia（墨尔本大学信息科技学院数据科学与人工智能系）；Nuffield Department of Clinical Neurosciences, University of Oxford, London, UK（牛津大学临床神经科学系）；School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China（电子科技大学机械与电子工程学院）；Department of Radiology, Sichuan Provincial People’s Hospital, Chengdu, China（四川省人民医院放射科）；Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China（东南大学医学院中大医院放射科）；Shanghai AI Lab, Shanghai, China（上海人工智能实验室）；Sensetime Research, Shanghai, China（商汤研究）
The use of accurate scanning transmission electron microscopy (STEM) image simulation methods require large computation times that can make their use infeasible for the simulation of many images. Other simulation methods based on linear imaging models, such as the convolution method, are much faster but are too inaccurate to be used in application. In this paper, we explore deep learning models that attempt to translate a STEM image produced by the convolution method to a prediction of the high accuracy multislice image. We then compare our results to those of regression methods. We find that …
Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applications
NCT/UCC Dresden（德累斯顿国家肿瘤疾病中心/德累斯顿肿瘤中心）；SECAI, TU Dresden（德累斯顿工业大学SECAI中心）；Faculty of medicine & University Hospital TU Dresden（德累斯顿工业大学医学院及大学医院）；CeTI, TU Dresden（德累斯顿工业大学CeTI中心）
Despite remarkable success in unpaired image-to-image translation, existing systems still require a large amount of labeled images. This is a bottleneck for their real-world applications; in practice, a model trained on labeled CelebA dataset does not work well for test images from a different distribution -- greatly limiting their application to unlabeled images of a much larger quantity. In this paper, we attempt to alleviate this necessity for labeled data in the facial image translation domain. We aim to explore the degree to which you can discover novel attributes from unlabeled faces and…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Siemens Healthineers（西门子医疗）；Rutgers University（罗格斯大学）
The one-to-one mapping is necessary for many bidirectional image-to-image translation applications, such as MRI image synthesis as MRI images are unique to the patient. State-of-the-art approaches for image synthesis from domain X to domain Y learn a convolutional neural network that meticulously maps between the domains. A different network is typically implemented to map along the opposite direction, from Y to X. In this paper, we explore the possibility of only wielding one network for bi-directional image synthesis. In other words, such an autonomous learning network implements a self-inve…
Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching
Università Campus Bio-Medico di Roma（罗马 Campus Bio-Medico 大学）；Umeå University（于默奥大学）；UniCamillus – Saint Camillus International University of Health Sciences（UniCamillus – 圣卡米勒斯国际健康科学大学）
Whole gland (WG) segmentation of the prostate plays a crucial role in detection, staging and treatment planning of prostate cancer (PCa). Despite promise shown by deep learning (DL) methods, they rely on the availability of a considerable amount of annotated data. Augmentation techniques such as translation and rotation of images present an alternative to increase data availability. Nevertheless, the amount of information provided by the transformed data is limited due to the correlation between the generated data and the original. Based on the recent success of generative adversarial networks…
Translating MRI to PET through Conditional Diffusion Models with Enhanced Pathology Awareness
Lab for Artificial Intelligence in Medical Imaging, Institute for Diagnostic and Interventional Radiology, School of Medicine and Health, TUM Klinikum, Technical University of Munich (TUM)（人工智能医学成像实验室，诊断与介入放射学研究所，医学院与健康学院，TUM医院，慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Nuclear Medicine, School of Medicine and Health（核医学系，医学院与健康学院）；Department of Neuroradiology, School of Medicine and Health（神经放射学系，医学院与健康学院）
Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-to-video translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames.We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal…
VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation
Center Algoritmi / LASI, University of Minho（中心Algoritmi / LASI，明霍大学）；Institute for Artificial Intelligence in Medicine (IKIM)（医学人工智能研究所）；Essen University Hospital (AöR)（埃森大学医院（AöR））；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心（CCCE））；West German Cancer Center（西德癌症中心）；University Hospital Essen (AöR)（埃森大学医院（AöR））；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；Partner site University Hospital Essen (AöR)（埃森大学医院（AöR）合作站点）；Institute of Computer Graphics and Vision（计算机图形与视觉研究所）；Graz University of Technology（格拉茨技术大学）；Institute of Medical Informatics（医学信息学研究所）；University Hospital RWTH Aachen（亚琛RWTH大学医院）；Department of Oral and Maxillofacial Surgery（口腔和颌面外科部门）；TU Dortmund University（多特蒙德技术大学）；Center for Virtual and Extended Reality in Medicine (ZvRM)（医学虚拟和扩展现实中心（ZvRM））；Faculty of Computer Science（计算机科学学院）
Multi-modal MRI Translation via Evidential Regression and Distribution Calibration
Fudan University（复旦大学）；University of Cambridge（剑桥大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Fuzhou University（福州市大学）；Shanghai Jiao Tong University（上海交通大学）
Guangdong University of Technology（广东工业大学）；The University of Sydney（悉尼大学）
Unpaired image-to-image translation aims to find a mapping between the source domain and the target domain. To alleviate the problem of the lack of supervised labels for the source images, cycle-consistency based methods have been proposed for image structure preservation by assuming a reversible relationship between unpaired images. However, this assumption only uses limited correspondence between image pairs. Recently, contrastive learning (CL) has been used to further investigate the image correspondence in unpaired image translation by using patch-based positive/negative learning. Patch-ba…
Exploring Siamese Networks in Self-Supervised Fast MRI Reconstruction
Stanford University School of Medicine（斯坦福大学医学院）；Xiamen University（厦门大学）；School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与工程学院）；School of Informatics, Xiamen University（厦门大学信息学院）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Stanford University（斯坦福大学）；GE Healthcare（通用电气医疗集团）；University of California Riverside（加州大学河滨分校）；National Taiwan University（台湾大学）
Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of cerebrovascular diseases such as Moyamoya, carotid stenosis, aneurysms, and stroke. Positron emission tomography (PET) is currently regarded as the gold standard for the measurement of CBF in the human brain. PET imaging, however, is not widely available because of its prohibitive costs, use of ionizing radiation, and logistical challenges, which require a co-localized cyclotron to deliver the 2 min half-life Oxygen-15 radioisotope. Magnetic resonance imaging (MRI), in contrast, is more readil…
Unsupervised domain adaptation is a type of domain adaptation and exploits labeled data from the source domain and unlabeled data from the target one. In the Cross-Modality Domain Adaptation for Medical Image Segmenta-tion challenge (crossMoDA2022), contrast enhanced T1 MRI volumes for brain are provided as the source domain data, and high-resolution T2 MRI volumes are provided as the target domain data. The crossMoDA2022 challenge contains two tasks, segmentation of vestibular schwannoma (VS) and cochlea, and clas-sification of VS with Koos grade. In this report, we presented our solution for…
Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Our contribution is a unified cross-modality feature disentagling approach for multi-domain image translation and multiple organ segmentation. Using CT as the labeled source domain, our approach learns to segment multi-modal (T1-weighted and T2-weighted) MRI having no labeled data. Our approach uses a variational auto-encoder (VAE) to disentangle the image content from style. The VAE constrains the style feature encoding to match a universal prior (Gaussian) that is assumed to span the styles of all the source and target modalities. The extracted image style is converted into a latent style sc…
