Automatically generating one medical imaging modality from another is known as medical image translation, and has numerous interesting applications. This paper presents an interpretable generative modelling approach to medical image translation. By allowing a common model for group-wise normalisation and segmentation of brain scans to handle missing data, the model allows for predicting entirely missing modalities from one, or a few, MR contrasts. Furthermore, the model can be trained on a fairly small number of subjects. The proposed model is validated on three clinically relevant scenarios. Results appear promising and show that a principled, probabilistic model of the relationship between multi-channel signal intensities can be used to infer missing modalities -- both MR contrasts and CT images.
Medical image translation has the potential to reduce the imaging workload, by removing the need to capture some sequences, and to reduce the annotation burden for developing machine learning methods. GANs have been used successfully to translate images from one domain to another, such as MR to CT. At present, paired data (registered MR and CT images) or extra supervision (e.g. segmentation masks) is needed to learn good translation models. Registering multiple modalities or annotating structures within each of them is a tedious and laborious task. Thus, there is a need to develop improved tra…
National Institutes of Health（美国国立卫生研究院）；University of Wisconsin（威斯康星大学）
Current deep learning based segmentation models often generalize poorly between domains due to insufficient training data. In real-world clinical applications, cross-domain image analysis tools are in high demand since medical images from different domains are often needed to achieve a precise diagnosis. An important example in radiology is generalizing from non-contrast CT to contrast enhanced CTs. Contrast enhanced CT scans at different phases are used to enhance certain pathologies or organs. Many existing cross-domain image-to-image translation models have been shown to improve cross-domai…
University of Stuttgart（斯图加特大学）；University of Tübingen（蒂宾根大学）；King’s College London（伦敦国王学院）
Image-to-image translation is a new field in computer vision with multiple potential applications in the medical domain. However, for supervised image translation frameworks, co-registered datasets, paired in a pixel-wise sense, are required. This is often difficult to acquire in realistic medical scenarios. On the other hand, unsupervised translation frameworks often result in blurred translated images with unrealistic details. In this work, we propose a new unsupervised translation framework which is titled Cycle-MedGAN. The proposed framework utilizes new non-adversarial cycle losses which …
Johns Hopkins University（约翰斯·霍普金斯大学）；Sigma Technologies（12 Sigma科技公司）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
Medical images are increasingly used as input to deep neural networks to produce quantitative values that aid researchers and clinicians. However, standard deep neural networks do not provide a reliable measure of uncertainty in those quantitative values. Recent work has shown that using dropout during training and testing can provide estimates of uncertainty. In this work, we investigate using dropout to estimate epistemic and aleatoric uncertainty in a CT-to-MR image translation task. We show that both types of uncertainty are captured, as defined, providing confidence in the output uncertai…
While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, it is oftentimes very difficult to obtain a lot of labeled medical images for model training. In this paper, we study cross-modality data augmentation to mitigate the data deficiency issue in the medical imaging domain. We propose a discriminative unpaired image-to-image translation model which translates images in source modality into images in target modality where the translation task is condu…
We cast the problem of image denoising as a domain translation problem between high and low noise domains. By modifying the cycleGAN model, we are able to learn a mapping between these domains on unpaired retinal optical coherence tomography images. In quantitative measurements and a qualitative evaluation by ophthalmologists, we show how this approach outperforms other established methods. The results indicate that the network differentiates subtle changes in the level of noise in the image. Further investigation of the model's feature maps reveals that it has learned to distinguish retinal l…
Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯大学医院）
The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the informat…
Self-Consistent Recursive Diffusion Bridge for Medical Image Translation
Bilkent University（毕尔肯大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心）；Stanford University（斯坦福大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Ambient-Pix2PixGAN for Translating Medical Images from Noisy Data
University of Michigan-Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术全球研究院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
pyMEAL: A Multi-Encoder Augmentation-Aware-Learning Toolbox for Robust Medical Image Translation
Hong Kong Centre for Cerebro-Cardiovascular Health Engineering (COCHE)（香港脑心血管健康工程中心）；Department of Informatics, Sokoine University of Agriculture（农业科学学院信息系）；Department of Diagnostic Radiology, The University of Hong Kong（香港大学诊断放射科）；State Key Laboratory of Brain and Cognitive Sciences, The University of Hong Kong（香港大学脑科学与认知科学国家重点实验室）；Alzheimer’s Disease Research Network, The University of Hong Kong（香港大学阿尔茨海默病研究网络）；Department of Biomedical Engineering, City University of Hong Kong（城市大学生物医学工程系）；Department of Radiology and Radiological Science, The Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院放射学与放射科学系）；City University of Hong Kong Shenzhen Research Institute（城市大学深圳研究 institute）
University of Queensland（昆士兰大学）；The Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织澳大利亚电子健康研究中心）；The University of Newcastle（纽卡斯尔大学）
Medical image translation (e.g. CT to MR) is a challenging task as it requires I) faithful translation of domain-invariant features (e.g. shape information of anatomical structures) and II) realistic synthesis of target-domain features (e.g. tissue appearance in MR). In this work, we propose Manifold Disentanglement Generative Adversarial Network (MDGAN), a novel image translation framework that explicitly models these two types of features. It employs a fully convolutional generator to model domain-invariant features, and it uses style codes to separately model target-domain features as a man…
COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training
Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Harvard Medical School and the Brigham and Women’s Hospital（哈佛医学院和布里洛妇产科医院）；MIND Team, Inria Saclay, Université Paris-Saclay, Palaiseau, France and Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（MIND团队、Inria Saclay、巴黎萨克雷大学、Palaiseau，法国以及索邦大学、巴黎脑研究所-ICM、CNRS、Inria、Inserm、AP-HP、皮蒂埃萨勒普蒂耶医院）；Massachusetts Institute of Technology（麻省理工学院）
GenMix: Combining Generative and Mixture Data Augmentation for Medical Image Classification
School of Electrical Engineering（电气工程学院）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Department of Software Convergence（软件融合系）；Seoul Women’s University（首尔女子大学）
Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Macao Polytechnic University（澳门理工大学）
Curvilinear Structure-preserving Unpaired Cross-domain Medical Image Translation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；Singapore Eye Research Institute, Singapore National Eye Centre, Singapore（新加坡眼科学研究所，新加坡国家眼科中心）；School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China（武汉科技大学计算机科学与技术学院）；Ophthalmology & Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore（眼科与视觉科学学术临床计划，新加坡国立大学医学院）；Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland（分子与临床眼科学研究所，瑞士巴塞尔）；School of Chemical and Biomedical Engineering, Nanyang Technological University, Singapore（南洋理工大学化学与生物医学工程学院）；Centre for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria（医学物理与生物医学工程中心，维也纳医科大学，维也纳，奥地利）；Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria（临床药理学系，维也纳医科大学，维也纳，奥地利）；Rothschild Foundation Hospital, Paris, France（罗特希尔德基金会医院，法国巴黎）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究所以（I 2 R），新加坡A*STAR）
MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation
Faculty of Computer Science and Information Technology, Universiti Malaya（马来亚大学计算机科学与信息学院）；School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算机与信息学院）
Arizona State University（亚利桑那州立大学）；ASU-Mayo Center for Innovative Imaging（亚利桑那州立大学-梅奥创新成像中心）；Mayo Clinic（梅奥诊所）
Automated anomaly detection from medical images, such as MRIs and X-rays, can significantly reduce human effort in disease diagnosis. Owing to the complexity of modeling anomalies and the high cost of manual annotation by domain experts (e.g., radiologists), a typical technique in the current medical imaging literature has focused on deriving diagnostic models from healthy subjects only, assuming the model will detect the images from patients as outliers. However, in many real-world scenarios, unannotated datasets with a mix of both healthy and diseased individuals are abundant. Therefore, thi…
Computer Vision Institute, Shenzhen University（深圳大学计算机视觉研究所）；Tencent Jarvis Lab（腾讯贾维斯实验室）
Domain shift between medical images from multicentres is still an open question for the community, which degrades the generalization performance of deep learning models. Generative adversarial network (GAN), which synthesize plausible images, is one of the potential solutions to address the problem. However, the existing GAN-based approaches are prone to fail at preserving image-objects in image-to-image (I2I) translation, which reduces their practicality on domain adaptation tasks. In this paper, we propose a novel GAN (namely MI$^2$GAN) to maintain image-contents during cross-domain I2I tran…
Attentive Continuous Generative Self-training for Unsupervised Domain Adaptive Medical Image Translation
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland School of Dentistry（马里兰大学牙医学院）
Multi-modal MRI Translation via Evidential Regression and Distribution Calibration
Fudan University（复旦大学）；University of Cambridge（剑桥大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Fuzhou University（福州市大学）；Shanghai Jiao Tong University（上海交通大学）
HiFi-Syn: Hierarchical Granularity Discrimination for High-Fidelity Synthesis of MR Images with Structure Preservation
Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学技术学院）；Clinical Neuroscience Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院临床神经科学中心）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy, Shanghai Jiao Tong University（上海交通大学先进磁共振诊断与治疗技术国家工程研究中心）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
ContourDiff: Unpaired Medical Image Translation with Structural Consistency
Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）；Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）；Department of Radiology, Duke University（杜克大学放射学系）；Department of Computer Science, Duke University（杜克大学计算机科学系）
Diffusion Bridge Models for 3D Medical Image Translation
University of California Riverside（加州大学河滨分校）；Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California（加州大学洛杉矶分校医学学院）；Information Sciences Institute, University of Southern California（加州大学洛杉矶分校信息科学研究所）
Local deformations in medical modalities are common phenomena due to a multitude of factors such as metallic implants or limited field of views in magnetic resonance imaging (MRI). Completion of the missing or distorted regions is of special interest for automatic image analysis frameworks to enhance post-processing tasks such as segmentation or classification. In this work, we propose a new generative framework for medical image inpainting, titled ipA-MedGAN. It bypasses the limitations of previous frameworks by enabling inpainting of arbitrary shaped regions without a prior localization of t…
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
We propose a framework for rotation and translation covariant deep learning using $SE(2)$ group convolutions. The group product of the special Euclidean motion group $SE(2)$ describes how a concatenation of two roto-translations results in a net roto-translation. We encode this geometric structure into convolutional neural networks (CNNs) via $SE(2)$ group convolutional layers, which fit into the standard 2D CNN framework, and which allow to generically deal with rotated input samples without the need for data augmentation. We introduce three layers: a lifting layer which lifts a 2D (vector va…
University of Toronto（多伦多大学）；Sunnybrook Research Institute（桑尼布鲁克研究所）
Quantitative medical image computing (radiomics) has been widely applied to build prediction models from medical images. However, overfitting is a significant issue in conventional radiomics, where a large number of radiomic features are directly used to train and test models that predict genotypes or clinical outcomes. In order to tackle this problem, we propose an unsupervised learning pipeline composed of an autoencoder for representation learning of radiomic features and a Gaussian mixture model based on minimum message length criterion for clustering. By incorporating probabilistic modeli…
Do We Really Need Diffusion? A Fast U-Net for Paired Medical Image Translation
Institute of Artificial Intelligence, Ulm University（乌尔姆大学人工智能研究所）；Institute of Orthopaedic Research and Biomechanics, Centre for Trauma Research, University Hospital Ulm（乌尔姆大学医院创伤研究中心骨科研究与生物力学研究所）；AI for Sensor Data Analytics Research Group, Ulm University of Applied Sciences（乌尔姆应用科学大学传感器数据分析人工智能研究组）
Information Bottleneck-based Causal Attention for Multi-label Medical Image Recognition
Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR)（联合山东大学-东北大学人工智能研究中心（C-FAIR））；Shandong University（山东大学）；School of Information Science and Engineering（信息科学与工程学院）；Child Care Hospital Affiliated to Shandong First Medical University（山东第一医科大学附属儿童医院）；Case Western Reserve University（凯斯西储大学）
Translation Consistent Semi-supervised Segmentation for 3D Medical Images
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Harvard Ophtalmology AI Lab of Schepens Eye Research Institute, Harvard Medical School（哈佛施佩尔眼科研究所眼科人工智能实验室，哈佛医学院）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息学院）；School of Electrical and Computer Engineering, Cornell University and Cornell Tech（康奈尔大学电气与计算机工程学院及康奈尔科技）；Faculty of Engineering of the Friedrich-Alexander-Universität Erlangen-Nürnberg（埃朗根-纽伦堡弗里德里希-亚历山大大学工程学院）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（视觉、语音和信号处理中心（CVSSP）， Surrey大学）
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Paired multi-modality medical images, can provide complementary information to help physicians make more reasonable decisions than single modality medical images. But they are difficult to generate due to multiple factors in practice (e.g., time, cost, radiation dose). To address these problems, multi-modality medical image translation has aroused increasing research interest recently. However, the existing works mainly focus on translation effect of a whole image instead of a critical target area or Region of Interest (ROI), e.g., organ and so on. This leads to poor-quality translation of the…
SRUG: A Fusion-Driven Generator Network for Medical Image Translation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（上海工程技术大学电子电气工程学院）；Department of Thoracic Surgery, Zhongshan Hospital of Fudan University（复旦大学中山医院胸外科）；Clinical Research Unit, Institute of Clinical Science, Zhongshan Hospital of Fudan University（复旦大学中山医院临床科研部）；Department of Medical Oncology, Cancer Center, and Fudan Zhangjiang Institute, Zhongshan Hospital of Fudan University（复旦大学中山医院医学肿瘤科、癌症中心及复旦张江研究院）
Cascaded Multi-path Shortcut Diffusion Model for Medical Image Translation
Yale University（耶鲁大学）；University of California Irvine（加州大学欧文分校）；University of Science and Technology of China（中国科学技术大学）；Case Western Reserve University（凯斯西储大学）；Northwestern University（西北大学）；Yale School of Medicine（耶鲁医学院）
Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, …
