Image synthesis from corrupted contrasts increases the diversity of diagnostic information available for many neurological diseases. Recently the image-to-image translation has experienced significant levels of interest within medical research, beginning with the successful use of the Generative Adversarial Network (GAN) to the introduction of cyclic constraint extended to multiple domains. However, in current approaches, there is no guarantee that the mapping between the two image domains would be unique or one-to-one. In this paper, we introduce a novel approach to unpaired image-to-image translation based on the invertible architecture. The invertible property of the flow-based architecture assures a cycle-consistency of image-to-image translation without additional loss functions. We utilize the temporal information between consecutive slices to provide more constraints to the optimization for transforming one domain to another in unpaired volumetric medical images. To capture temporal structures in the medical images, we explore the displacement between the consecutive slices using a deformation field. In our approach, the deformation field is used as a guidance to keep the translated slides realistic and consistent across the translation. The experimental results have shown that the synthesized images using our proposed approach are able to archive a competitive performance in terms of mean squared error, peak signal-to-noise ratio, and structural similarity index when compared with the existing deep learning-based methods on three standard datasets, i.e. HCP, MRBrainS13, and Brats2019.
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））；Seoul National University College of Medicine, Seoul National University Bundang Hospital（首尔大学医学院，首尔大学盆唐医院）
Recently, deep learning approaches for accelerated MRI have been extensively studied thanks to their high performance reconstruction in spite of significantly reduced runtime complexity. These neural networks are usually trained in a supervised manner, so matched pairs of subsampled and fully sampled k-space data are required. Unfortunately, it is often difficult to acquire matched fully sampled k-space data, since the acquisition of fully sampled k-space data requires long scan time and often leads to the change of the acquisition protocol. Therefore, unpaired deep learning without matched la…
Lack of ground-truth MR images impedes the common supervised training of neural networks for image reconstruction. To cope with this challenge, this paper leverages unpaired adversarial training for reconstruction networks, where the inputs are undersampled k-space and naively reconstructed images from one dataset, and the labels are high-quality images from another dataset. The reconstruction networks consist of a generator which suppresses the input image artifacts, and a discriminator using a pool of (unpaired) labels to adjust the reconstruction quality. The generator is an unrolled neural…
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Osaka University（大阪大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
CT is commonly used in orthopedic procedures. MRI is used along with CT to identify muscle structures and diagnose osteonecrosis due to its superior soft tissue contrast. However, MRI has poor contrast for bone structures. Clearly, it would be helpful if a corresponding CT were available, as bone boundaries are more clearly seen and CT has standardized (i.e., Hounsfield) units. Therefore, we aim at MR-to-CT synthesis. The CycleGAN was successfully applied to unpaired CT and MR images of the head, these images do not have as much variation of intensity pairs as do images in the pelvic region du…
Unpaired Object-Level SAR-to-Optical Image Translation for Aircraft with Keypoints-Guided Diffusion Models
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Key Laboratory for Information Science of Electromagnetic Waves (Ministry of Education)（电磁波信息科学教育部重点实验室）
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Magnetic Resonance (MR) Imaging and Computed Tomography (CT) are the primary diagnostic imaging modalities quite frequently used for surgical planning and analysis. A general problem with medical imaging is that the acquisition process is quite expensive and time-consuming. Deep learning techniques like generative adversarial networks (GANs) can help us to leverage the possibility of an image to image translation between multiple imaging modalities, which in turn helps in saving time and cost. These techniques will help to conduct surgical planning under CT with the feedback of MRI information…
Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utili…
Segmenting the fine structure of the mouse brain on magnetic resonance (MR) images is critical for delineating morphological regions, analyzing brain function, and understanding their relationships. Compared to a single MRI modality, multimodal MRI data provide complementary tissue features that can be exploited by deep learning models, resulting in better segmentation results. However, multimodal mouse brain MRI data is often lacking, making automatic segmentation of mouse brain fine structure a very challenging task. To address this issue, it is necessary to fuse multimodal MRI data to produ…
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（右伊萨尔医院）；University of Zurich（苏黎世大学）；Institut für KI und Informatik in der Medizin（医学人工智能与信息学研究所）；Imperial College London（伦敦帝国学院）
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Accurate and robust segmentation of lung cancers from CT, even those located close to mediastinum, is needed to more accurately plan and deliver radiotherapy and to measure treatment response. Therefore, we developed a new cross-modality educed distillation (CMEDL) approach, using unpaired CT and MRI scans, whereby an informative teacher MRI network guides a student CT network to extract features that signal the difference between foreground and background. Our contribution eliminates two requirements of distillation methods: (i) paired image sets by using an image to image (I2I) translation a…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）
We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-networ…
Retrospective artifact correction (RAC) improves image quality post acquisition and enhances image usability. Recent machine learning driven techniques for RAC are predominantly based on supervised learning and therefore practical utility can be limited as data with paired artifact-free and artifact-corrupted images are typically insufficient or even non-existent. Here we show that unwanted image artifacts can be disentangled and removed from an image via an RAC neural network learned with unpaired data. This implies that our method does not require matching artifact-corrupted data to be eithe…
Anatomy-preserving unpaired cone-beam CT refinement for image-guided radiotherapy using pseudo-label guided diffusion
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Southern University of Science and Technology（南方科技大学）；Huashan Hospital, Fudan University（复旦大学附属华山医院）
MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction
1 Hawkes Institute, UCL \, 2 Dept. of Medical Physics；Biomedical Engineering, UCL 3 Dept. of Computer Science, UCL 4 Centre for AI, DS\&AI, AstraZeneca, UK
Auxiliary CycleGAN-guidance for Task-Aware Domain Translation from Duplex to Monoplex IHC Images
AstraZeneca Computational Pathology, Oncology Research and Development（阿斯利康肿瘤研发部计算病理学）；AstraZeneca Translational Pathology, Oncology Research and Development（阿斯利康肿瘤研发部转化病理学）
Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization
University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）；Global Institute of Future Technology, Shanghai Jiao Tong University（上海交通大学未来技术学院）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）
ModaFlow: Modality-Aware Flow Matching for High-Fidelity Virtual Try-On
South China University of Technology（华南理工大学）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis（广东省多模态大数据智能分析重点实验室）
A Two-Stage Generative Model with CycleGAN and Joint Diffusion for MRI-based Brain Tumor Detection
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University Shenzhen Hospital（北京大学深圳医院）；National Center for Applied Mathematics Shenzhen（深圳国家应用数学中心）
Fetal brain magnetic resonance imaging (MRI) offers exquisite images of the developing brain but is not suitable for anomaly screening. For this ultrasound (US) is employed. While expert sonographers are adept at reading US images, MR images are much easier for non-experts to interpret. Hence in this paper we seek to produce images with MRI-like appearance directly from clinical US images. Our own clinical motivation is to seek a way to communicate US findings to patients or clinical professionals unfamiliar with US, but in medical image analysis such a capability is potentially useful, for in…
Deformable Image Registration (DIR) of MR and CT images is one of the most challenging registration task, due to the inherent structural differences of the modalities and the missing dense ground truth. Recently cycle Generative Adversarial Networks (cycle-GANs) have been used to learn the intensity relationship between these 2 modalities for unpaired brain data. Yet its usefulness for DIR was not assessed. In this study we evaluate the DIR performance for thoracic and abdominal organs after synthesis by cycle-GAN. We show that geometric changes, which differentiate the two populations (e.g. i…
Generative adversarial networks (GANs) are currently rarely applied on 3D medical images of large size, due to their immense computational demand. The present work proposes a multi-scale patch-based GAN approach for establishing unpaired domain translation by generating 3D medical image volumes of high resolution in a memory-efficient way. The key idea to enable memory-efficient image generation is to first generate a low-resolution version of the image followed by the generation of patches of constant sizes but successively growing resolutions. To avoid patch artifacts and incorporate global …
Dual-domain Multi-path Self-supervised Diffusion Model for Accelerated MRI Reconstruction
Department of Radiology, Northwestern University（放射科，西北大学）；Department of Biomedical Engineering, Huazhong University of Science and Technology（生物医学工程系，华中科技大学）
Trustworthy Longitudinal Brain MRI Completion: A Deformation-Based Approach with KAN-Enhanced Diffusion Model
School of Biomedical Engineering and Imaging Science, King's College London, London, UK（生物医学工程与成像科学学院，伦敦国王学院）；School of Biomedical Engineering, ShanghaiTech University, Shanghai, China（生物医学工程学院，上海科技大学）；School of Computer Science and Digital Technologies, Aston University, Birmingham, UK（计算机科学与数字技术学院，阿斯顿大学）；Kunming Medical University, Kunming, China（昆明医学院）
Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（拉特伯研究中心生物医学成像，深圳先进技术研究所，中国科学院）；School of Artificial Intelligence, the University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Research Center for Medical AI, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（医学人工智能研究中心，深圳先进技术研究所，中国科学院）；Pazhou Lab, Guangzhou, China（琶洲实验室，广州，中国）；School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China（生物医学工程学院，深圳大学医学院，深圳大学，深圳，中国）
A lack of generalizability is one key limitation of deep learning based segmentation. Typically, one manually labels new training images when segmenting organs in different imaging modalities or segmenting abnormal organs from distinct disease cohorts. The manual efforts can be alleviated if one is able to reuse manual labels from one modality (e.g., MRI) to train a segmentation network for a new modality (e.g., CT). Previously, two stage methods have been proposed to use cycle generative adversarial networks (CycleGAN) to synthesize training images for a target modality. Then, these efforts t…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Incyte Corporation（Incyte公司）
A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality with…
Diffusion models have achieved unprecedented performance in generative modeling. The commonly-adopted formulation of the latent code of diffusion models is a sequence of gradually denoised samples, as opposed to the simpler (e.g., Gaussian) latent space of GANs, VAEs, and normalizing flows. This paper provides an alternative, Gaussian formulation of the latent space of various diffusion models, as well as an invertible DPM-Encoder that maps images into the latent space. While our formulation is purely based on the definition of diffusion models, we demonstrate several intriguing consequences. …
Anguinus Sculpturae: Compositional Synthesis of Peak-Enhancement Breast DCE-MRI Scans
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；HIDSS4Health – Helmholtz Information and Data Science School for Health（HIDSS4Health – 亥姆霍兹健康信息与数据科学学校）；Helmholtz Imaging（亥姆霍兹成像中心）
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland（马里兰大学）
Understanding the underlying relationship between tongue and oropharyngeal muscle deformation seen in tagged-MRI and intelligible speech plays an important role in advancing speech motor control theories and treatment of speech related-disorders. Because of their heterogeneous representations, however, direct mapping between the two modalities -- i.e., two-dimensional (mid-sagittal slice) plus time tagged-MRI sequence and its corresponding one-dimensional waveform -- is not straightforward. Instead, we resort to two-dimensional spectrograms as an intermediate representation, which contains bot…
New York University Tandon School of Engineering（纽约大学坦登工程学院）
Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which l…
Indian Institute of Technology Madras (IITM)（印度理工学院马德拉斯分校）；Healthcare Technology Innovation Centre (HTIC), IITM（印度理工学院马德拉斯分校医疗技术创新中心）
Magnetic Resonance Imaging (MRI) is a valuable clinical diagnostic modality for spine pathologies with excellent characterization for infection, tumor, degenerations, fractures and herniations. However in surgery, image-guided spinal procedures continue to rely on CT and fluoroscopy, as MRI slice resolutions are typically insufficient. Building upon state-of-the-art single image super-resolution, we propose a reference-based, unpaired multi-contrast texture-transfer strategy for deep learning based in-plane and across-plane MRI super-resolution. We use the scattering transform to relate the te…
Imperial College London（伦敦帝国理工学院）；The Chinese University of Hong Kong（香港中文大学）
Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model,…
