Medical images are increasingly used as input to deep neural networks to produce quantitative values that aid researchers and clinicians. However, standard deep neural networks do not provide a reliable measure of uncertainty in those quantitative values. Recent work has shown that using dropout during training and testing can provide estimates of uncertainty. In this work, we investigate using dropout to estimate epistemic and aleatoric uncertainty in a CT-to-MR image translation task. We show that both types of uncertainty are captured, as defined, providing confidence in the output uncertainty estimates.
While deep learning methods have shown great success in medical image analysis, they require a number of medical images to train. Due to data privacy concerns and unavailability of medical annotators, it is oftentimes very difficult to obtain a lot of labeled medical images for model training. In this paper, we study cross-modality data augmentation to mitigate the data deficiency issue in the medical imaging domain. We propose a discriminative unpaired image-to-image translation model which translates images in source modality into images in target modality where the translation task is condu…
We cast the problem of image denoising as a domain translation problem between high and low noise domains. By modifying the cycleGAN model, we are able to learn a mapping between these domains on unpaired retinal optical coherence tomography images. In quantitative measurements and a qualitative evaluation by ophthalmologists, we show how this approach outperforms other established methods. The results indicate that the network differentiates subtle changes in the level of noise in the image. Further investigation of the model's feature maps reveals that it has learned to distinguish retinal l…
Wellcome Centre for Human Neuroimaging（惠康人类神经影像中心）；UCL（伦敦大学学院）；UCL Institute of Neurology（伦敦大学学院神经学研究所）
Automatically generating one medical imaging modality from another is known as medical image translation, and has numerous interesting applications. This paper presents an interpretable generative modelling approach to medical image translation. By allowing a common model for group-wise normalisation and segmentation of brain scans to handle missing data, the model allows for predicting entirely missing modalities from one, or a few, MR contrasts. Furthermore, the model can be trained on a fairly small number of subjects. The proposed model is validated on three clinically relevant scenarios. …
University of Stuttgart（斯图加特大学）；University of Tübingen（蒂宾根大学）；King’s College London（伦敦国王学院）
Image-to-image translation is a new field in computer vision with multiple potential applications in the medical domain. However, for supervised image translation frameworks, co-registered datasets, paired in a pixel-wise sense, are required. This is often difficult to acquire in realistic medical scenarios. On the other hand, unsupervised translation frameworks often result in blurred translated images with unrealistic details. In this work, we propose a new unsupervised translation framework which is titled Cycle-MedGAN. The proposed framework utilizes new non-adversarial cycle losses which …
Medical image translation has the potential to reduce the imaging workload, by removing the need to capture some sequences, and to reduce the annotation burden for developing machine learning methods. GANs have been used successfully to translate images from one domain to another, such as MR to CT. At present, paired data (registered MR and CT images) or extra supervision (e.g. segmentation masks) is needed to learn good translation models. Registering multiple modalities or annotating structures within each of them is a tedious and laborious task. Thus, there is a need to develop improved tra…
National Institutes of Health（美国国立卫生研究院）；University of Wisconsin（威斯康星大学）
Current deep learning based segmentation models often generalize poorly between domains due to insufficient training data. In real-world clinical applications, cross-domain image analysis tools are in high demand since medical images from different domains are often needed to achieve a precise diagnosis. An important example in radiology is generalizing from non-contrast CT to contrast enhanced CTs. Contrast enhanced CT scans at different phases are used to enhance certain pathologies or organs. Many existing cross-domain image-to-image translation models have been shown to improve cross-domai…
Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯大学医院）
The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the informat…
Self-Consistent Recursive Diffusion Bridge for Medical Image Translation
Bilkent University（毕尔肯大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心）；Stanford University（斯坦福大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Ambient-Pix2PixGAN for Translating Medical Images from Noisy Data
University of Michigan-Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术全球研究院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
University of Queensland（昆士兰大学）；The Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织澳大利亚电子健康研究中心）；The University of Newcastle（纽卡斯尔大学）
Medical image translation (e.g. CT to MR) is a challenging task as it requires I) faithful translation of domain-invariant features (e.g. shape information of anatomical structures) and II) realistic synthesis of target-domain features (e.g. tissue appearance in MR). In this work, we propose Manifold Disentanglement Generative Adversarial Network (MDGAN), a novel image translation framework that explicitly models these two types of features. It employs a fully convolutional generator to model domain-invariant features, and it uses style codes to separately model target-domain features as a man…
pyMEAL: A Multi-Encoder Augmentation-Aware-Learning Toolbox for Robust Medical Image Translation
Hong Kong Centre for Cerebro-Cardiovascular Health Engineering (COCHE)（香港脑心血管健康工程中心）；Department of Informatics, Sokoine University of Agriculture（农业科学学院信息系）；Department of Diagnostic Radiology, The University of Hong Kong（香港大学诊断放射科）；State Key Laboratory of Brain and Cognitive Sciences, The University of Hong Kong（香港大学脑科学与认知科学国家重点实验室）；Alzheimer’s Disease Research Network, The University of Hong Kong（香港大学阿尔茨海默病研究网络）；Department of Biomedical Engineering, City University of Hong Kong（城市大学生物医学工程系）；Department of Radiology and Radiological Science, The Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院放射学与放射科学系）；City University of Hong Kong Shenzhen Research Institute（城市大学深圳研究 institute）
ContourDiff: Unpaired Medical Image Translation with Structural Consistency
Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）；Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）；Department of Radiology, Duke University（杜克大学放射学系）；Department of Computer Science, Duke University（杜克大学计算机科学系）
Attentive Continuous Generative Self-training for Unsupervised Domain Adaptive Medical Image Translation
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland School of Dentistry（马里兰大学牙医学院）
We propose a framework for rotation and translation covariant deep learning using $SE(2)$ group convolutions. The group product of the special Euclidean motion group $SE(2)$ describes how a concatenation of two roto-translations results in a net roto-translation. We encode this geometric structure into convolutional neural networks (CNNs) via $SE(2)$ group convolutional layers, which fit into the standard 2D CNN framework, and which allow to generically deal with rotated input samples without the need for data augmentation. We introduce three layers: a lifting layer which lifts a 2D (vector va…
Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Macao Polytechnic University（澳门理工大学）
COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training
Curvilinear Structure-preserving Unpaired Cross-domain Medical Image Translation
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；Singapore Eye Research Institute, Singapore National Eye Centre, Singapore（新加坡眼科学研究所，新加坡国家眼科中心）；School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China（武汉科技大学计算机科学与技术学院）；Ophthalmology & Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore（眼科与视觉科学学术临床计划，新加坡国立大学医学院）；Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland（分子与临床眼科学研究所，瑞士巴塞尔）；School of Chemical and Biomedical Engineering, Nanyang Technological University, Singapore（南洋理工大学化学与生物医学工程学院）；Centre for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria（医学物理与生物医学工程中心，维也纳医科大学，维也纳，奥地利）；Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria（临床药理学系，维也纳医科大学，维也纳，奥地利）；Rothschild Foundation Hospital, Paris, France（罗特希尔德基金会医院，法国巴黎）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究所以（I 2 R），新加坡A*STAR）
Emory University（埃默里大学）；Indiana University Purdue University-Indianapolis（印第安纳大学与普渡大学印第安纳波利斯联合分校）
Deep learning approaches applied to medical imaging have reached near-human or better-than-human performance on many diagnostic tasks. For instance, the CheXpert competition on detecting pathologies in chest x-rays has shown excellent multi-class classification performance. However, training and validating deep learning models require extensive collections of images and still produce false inferences, as identified by a human-in-the-loop. In this paper, we introduce a practical approach to improve the predictions of a pre-trained model through Few-Shot Learning (FSL). After training and valida…
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
Arizona State University（亚利桑那州立大学）；ASU-Mayo Center for Innovative Imaging（亚利桑那州立大学-梅奥创新成像中心）；Mayo Clinic（梅奥诊所）
Automated anomaly detection from medical images, such as MRIs and X-rays, can significantly reduce human effort in disease diagnosis. Owing to the complexity of modeling anomalies and the high cost of manual annotation by domain experts (e.g., radiologists), a typical technique in the current medical imaging literature has focused on deriving diagnostic models from healthy subjects only, assuming the model will detect the images from patients as outliers. However, in many real-world scenarios, unannotated datasets with a mix of both healthy and diseased individuals are abundant. Therefore, thi…
Diffusion Bridge Models for 3D Medical Image Translation
University of California Riverside（加州大学河滨分校）；Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California（加州大学洛杉矶分校医学学院）；Information Sciences Institute, University of Southern California（加州大学洛杉矶分校信息科学研究所）
Computer Vision Institute, Shenzhen University（深圳大学计算机视觉研究所）；Tencent Jarvis Lab（腾讯贾维斯实验室）
Domain shift between medical images from multicentres is still an open question for the community, which degrades the generalization performance of deep learning models. Generative adversarial network (GAN), which synthesize plausible images, is one of the potential solutions to address the problem. However, the existing GAN-based approaches are prone to fail at preserving image-objects in image-to-image (I2I) translation, which reduces their practicality on domain adaptation tasks. In this paper, we propose a novel GAN (namely MI$^2$GAN) to maintain image-contents during cross-domain I2I tran…
Multi-modal MRI Translation via Evidential Regression and Distribution Calibration
Fudan University（复旦大学）；University of Cambridge（剑桥大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Fuzhou University（福州市大学）；Shanghai Jiao Tong University（上海交通大学）
HiFi-Syn: Hierarchical Granularity Discrimination for High-Fidelity Synthesis of MR Images with Structure Preservation
Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学技术学院）；Clinical Neuroscience Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院临床神经科学中心）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy, Shanghai Jiao Tong University（上海交通大学先进磁共振诊断与治疗技术国家工程研究中心）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
Local deformations in medical modalities are common phenomena due to a multitude of factors such as metallic implants or limited field of views in magnetic resonance imaging (MRI). Completion of the missing or distorted regions is of special interest for automatic image analysis frameworks to enhance post-processing tasks such as segmentation or classification. In this work, we propose a new generative framework for medical image inpainting, titled ipA-MedGAN. It bypasses the limitations of previous frameworks by enabling inpainting of arbitrary shaped regions without a prior localization of t…
Do We Really Need Diffusion? A Fast U-Net for Paired Medical Image Translation
Institute of Artificial Intelligence, Ulm University（乌尔姆大学人工智能研究所）；Institute of Orthopaedic Research and Biomechanics, Centre for Trauma Research, University Hospital Ulm（乌尔姆大学医院创伤研究中心骨科研究与生物力学研究所）；AI for Sensor Data Analytics Research Group, Ulm University of Applied Sciences（乌尔姆应用科学大学传感器数据分析人工智能研究组）
Translation Consistent Semi-supervised Segmentation for 3D Medical Images
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Harvard Ophtalmology AI Lab of Schepens Eye Research Institute, Harvard Medical School（哈佛施佩尔眼科研究所眼科人工智能实验室，哈佛医学院）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息学院）；School of Electrical and Computer Engineering, Cornell University and Cornell Tech（康奈尔大学电气与计算机工程学院及康奈尔科技）；Faculty of Engineering of the Friedrich-Alexander-Universität Erlangen-Nürnberg（埃朗根-纽伦堡弗里德里希-亚历山大大学工程学院）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（视觉、语音和信号处理中心（CVSSP）， Surrey大学）
PETWB-REP: A Multi-Cancer Whole-Body FDG PET/CT and Radiology Report Dataset for Medical Imaging Research
PET Center, Huashan Hospital, Fudan University（复旦大学附属华山医院PET中心）；Shanghai Academy of Artificial Intelligence for Science（上海科学智能研究院）；Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Shanghai Universal Medical Imaging Diagnostic Center（上海环球医学影像诊断中心）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Image-to-image translation plays a vital role in tackling various medical imaging tasks such as attenuation correction, motion correction, undersampled reconstruction, and denoising. Generative adversarial networks have been shown to achieve the state-of-the-art in generating high fidelity images for these tasks. However, the state-of-the-art GAN-based frameworks do not estimate the uncertainty in the predictions made by the network that is essential for making informed medical decisions and subsequent revision by medical experts and has recently been shown to improve the performance and inter…
SRUG: A Fusion-Driven Generator Network for Medical Image Translation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（上海工程技术大学电子电气工程学院）；Department of Thoracic Surgery, Zhongshan Hospital of Fudan University（复旦大学中山医院胸外科）；Clinical Research Unit, Institute of Clinical Science, Zhongshan Hospital of Fudan University（复旦大学中山医院临床科研部）；Department of Medical Oncology, Cancer Center, and Fudan Zhangjiang Institute, Zhongshan Hospital of Fudan University（复旦大学中山医院医学肿瘤科、癌症中心及复旦张江研究院）
Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Ulm University Medical Center（乌尔姆大学医学中心）；BGZ - Bildgebungszentrum（BGZ影像中心）；i2SouI - Innovative Imaging in Surgical Oncology Ulm（乌尔姆外科肿瘤创新影像中心）；XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology（XAIRAD实验放射学人工智能合作中心）
Cross Modality Image Translation In Medical Imaging Using Generative Frameworks
Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University（诊断与介入部门、放射物理、生物医学工程，乌梅大学）；Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma（人工智能与计算机系统单位，工程部门，罗马生物医学学院）；Vita-Salute San Raffaele University（维塔-萨拉特·桑拉法埃莱大学）；Department of Medicine, Surgery and Dentistry, University of Salerno（医学、外科和牙科部门，萨勒诺大学）；Division of Diagnostic and Interventional Neuroradiology, Department of Radiology, University Hospital Basel（诊断和介入神经放射学部门，放射学部门，巴塞尔大学医院）；Department of Pediatric Radiology, University Children’s Hospital Basel（儿科放射学部门，巴塞尔儿童医院）；Department of Life Science and Public Health, Università Cattolica del Sacro Cuore（生命科学与公共健康部门，圣心大学）；Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉A·马里诺斯生物医学成像中心）；Artificial Intelligence and Translational Imaging (ATI) Lab, Department of Radiology, School of Medicine, University of Crete（人工智能与转化成像（ATI）实验室，放射学部门，医学院，克里特大学）；Division of Radiology, Department of Clinical Science, Intervention and Technology (CLINTEC), Karolinska Institute（放射学部门，临床科学、介入和科技（CLINTEC）部门，卡罗林斯卡研究所）；Columbia University Medical Center（哥伦比亚大学医学中心）；Department of Diagnostics and intervention, Diagnostic radiology, Umeå University（诊断与介入部门，诊断放射学，乌梅大学）
Imperial College London（帝国理工学院）；TU München（慕尼黑工业大学）；FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）
Medical image analysis is a vibrant research area that offers doctors and medical practitioners invaluable insight and the ability to accurately diagnose and monitor disease. Machine learning provides an additional boost for this area. However, machine learning for medical image analysis is particularly vulnerable to natural biases like domain shifts that affect algorithmic performance and robustness. In this paper we analyze machine learning for medical image analysis within the framework of Technology Readiness Levels and review how causal analysis methods can fill a gap when creating robust…
Generative adversarial networks (GANs) are currently rarely applied on 3D medical images of large size, due to their immense computational demand. The present work proposes a multi-scale patch-based GAN approach for establishing unpaired domain translation by generating 3D medical image volumes of high resolution in a memory-efficient way. The key idea to enable memory-efficient image generation is to first generate a low-resolution version of the image followed by the generation of patches of constant sizes but successively growing resolutions. To avoid patch artifacts and incorporate global …
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Paired multi-modality medical images, can provide complementary information to help physicians make more reasonable decisions than single modality medical images. But they are difficult to generate due to multiple factors in practice (e.g., time, cost, radiation dose). To address these problems, multi-modality medical image translation has aroused increasing research interest recently. However, the existing works mainly focus on translation effect of a whole image instead of a critical target area or Region of Interest (ROI), e.g., organ and so on. This leads to poor-quality translation of the…
Cascaded Multi-path Shortcut Diffusion Model for Medical Image Translation
Yale University（耶鲁大学）；University of California Irvine（加州大学欧文分校）；University of Science and Technology of China（中国科学技术大学）；Case Western Reserve University（凯斯西储大学）；Northwestern University（西北大学）；Yale School of Medicine（耶鲁医学院）
Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, …
