Positron emission tomography (PET) imaging is an imaging modality for diagnosing a number of neurological diseases. In contrast to Magnetic Resonance Imaging (MRI), PET is costly and involves injecting a radioactive substance into the patient. Motivated by developments in modality transfer in vision, we study the generation of certain types of PET images from MRI data. We derive new flow-based generative models which we show perform well in this small sample size regime (much smaller than dataset sizes available in standard vision tasks). Our formulation, DUAL-GLOW, is based on two invertible networks and a relation network that maps the latent spaces to each other. We discuss how given the prior distribution, learning the conditional distribution of PET given the MRI image reduces to obtaining the conditional distribution between the two latent codes w.r.t. the two image types. We also extend our framework to leverage 'side' information (or attributes) when available. By controlling the PET generation through 'conditioning' on age, our model is also able to capture brain FDG-PET (hypometabolism) changes, as a function of age. We present experiments on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset with 826 subjects, and obtain good performance in PET image synthesis, qualitatively and quantitatively better than recent works.
KTH Royal Institute of Technology（瑞典皇家理工学院）；Silo AI（赛罗人工智能公司）
Autonomous agents, such as driverless cars, require large amounts of labeled visual data for their training. A viable approach for acquiring such data is training a generative model with collected real data, and then augmenting the collected real dataset with synthetic images from the model, generated with control of the scene layout and ground truth labeling. In this paper we propose Full-Glow, a fully conditional Glow-based architecture for generating plausible and realistic images of novel street scenes given a semantic segmentation map indicating the scene layout. Benchmark comparisons sho…
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology（华南理工大学）；JD Explore Academy（京东探索研究院）；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
University of Dundee（邓迪大学）；University of Cambridge（剑桥大学）；University of Bath（巴斯大学）；East China Normal University（华东师范大学）
MRI synthesis promises to mitigate the challenge of missing MRI modality in clinical practice. Diffusion model has emerged as an effective technique for image synthesis by modelling complex and variable data distributions. However, most diffusion-based MRI synthesis models are using a single modality. As they operate in the original image domain, they are memory-intensive and less feasible for multi-modal synthesis. Moreover, they often fail to preserve the anatomical structure in MRI. Further, balancing the multiple conditions from multi-modal MRI inputs is crucial for multi-modal synthesis. …
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China（中国科学院智能信息处理重点实验室（中国科学院计算技术研究所，中国））；University of Chinese Academy of Sciences, China（中国科学院大学）
Chinese University of Hong Kong（香港中文大学）；Texas A&M University（得克萨斯农工大学）
Flow-based generative models show great potential in image synthesis due to its reversible pipeline and exact log-likelihood target, yet it suffers from weak ability for conditional image synthesis, especially for multi-label or unaware conditions. This is because the potential distribution of image conditions is hard to measure precisely from its latent variable $z$. In this paper, based on modeling a joint probabilistic density of an image and its conditions, we propose a novel flow-based generative model named conditional adversarial generative flow (CAGlow). Instead of disentangling attrib…
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation
Nanyang Technological University（南洋理工大学）；South China University of Technology（华南理工大学）；University of Oxford（牛津大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
Plasma-CycleGAN: Plasma Biomarker-Guided MRI to PET Cross-modality Translation Using Conditional CycleGAN
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）；Basis Scottsdale（斯科茨代尔贝斯机构）；College of Health Solutions, Arizona State University（亚利桑那州立大学健康解决方案学院）；Mayo Clinic Arizona（梅奥诊所亚利桑那州分院）
As a powerful technique in medical imaging, image synthesis is widely used in applications such as denoising, super resolution and modality transformation etc. Recently, the revival of deep neural networks made immense progress in the field of medical imaging. Although many deep leaning based models have been proposed to improve the image synthesis accuracy, the evaluation of the model uncertainty, which is highly important for medical applications, has been a missing part. In this work, we propose to use Bayesian conditional generative adversarial network (GAN) with concrete dropout to improv…
National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）
The accurate prognosis of Glioblastoma Multiforme (GBM) plays an essential role in planning correlated surgeries and treatments. The conventional models of survival prediction rely on radiomic features using magnetic resonance imaging (MRI). In this paper, we propose a radiogenomic overall survival (OS) prediction approach by incorporating gene expression data with radiomic features such as shape, geometry, and clinical information. We exploit TCGA (The Cancer Genomic Atlas) dataset and synthesize the missing MRI modalities using a fully convolutional network (FCN) in a conditional Generative …
MMFusion: Multi-modality Diffusion Model for Lymph Node Metastasis Diagnosis in Esophageal Cancer
Shandong University（山东大学）；Hangzhou Dianzi University（杭州电子科技大学）；Communication University of Zhejiang（浙江传媒学院）；University of Leicester（莱斯特大学）；Sichuan Cancer Hospital and Institution（四川省肿瘤医院）；Sichuan Cancer Center（四川省癌症中心）；University of Electronic Science and Technology of China（电子科技大学）；Suzhou Research Institute of Shandong University（山东大学苏州研究院）
Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator…
OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model
Shanghai Key Laboratory of Intelligent Sensing and Recognition（上海智能感知与识别重点实验室）；Shanghai Jiao Tong University（上海交通大学）；Center of Digital Innovation（数字创新中心）
Research Institute of Beihang University in Shenzhen（北京航空航天大学深圳研究院）；Beihang University（北京航空航天大学）；Chongqing University（重庆大学）；Microsoft Research（微软研究院）
We present a cross-modality generation framework that learns to generate translated modalities from given modalities in MR images without real acquisition. Our proposed method performs NeuroImage-to-NeuroImage translation (abbreviated as N2N) by means of a deep learning model that leverages conditional generative adversarial networks (cGANs). Our framework jointly exploits the low-level features (pixel-wise information) and high-level representations (e.g. brain tumors, brain structure like gray matter, etc.) between cross modalities which are important for resolving the challenging complexity…
Massachusetts Eye and Ear（麻省眼耳医院）；Harvard Medical School（哈佛医学院）；Florida International University（佛罗里达国际大学）
Neuroimaging is essential in brain studies for the diagnosis and identification of disease, structure, and function of the brain in its healthy and disease states. Literature shows that there are advantages of multitasking with some deep learning (DL) schemes in challenging neuroimaging applications. This study examines the feasibility of using multitasking in three different applications, including tissue segmentation, cross-modality conversion, and bias-field correction. These applications reflect five different scenarios in which multitasking is explored and 280 training and testing session…
Domains such as logo synthesis, in which the data has a high degree of multi-modality, still pose a challenge for generative adversarial networks (GANs). Recent research shows that progressive training (ProGAN) and mapping network extensions (StyleGAN) enable both increased training stability for higher dimensional problems and better feature separation within the embedded latent space. However, these architectures leave limited control over shaping the output of the network, which is an undesirable trait in the case of logo synthesis. This paper explores a conditional extension to the StyleGA…
Wavelet-Fusion Diffusion Model for Multimodal Brain MRI Synthesis with Modality and Metadata Conditioning
Department of Psychology & Neuroscience, Auckland University of Technology（心理学与神经科学系，奥克兰技术大学）；Department of Psychiatry, University of Melbourne（精神病学系，墨尔本大学）
DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion
Xi’an Jiaotong University（西安交通大学）；Computer Vision Lab, ETH Zürich（苏黎世联邦理工学院计算机视觉实验室）；Northwestern Polytechnical University（西北工业大学）；Macau University of Science and Technology（澳门科技大学）；University of Würzburg（维尔茨堡大学）
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（机器人学国家重点实验室，中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Visual-tactile fused sensing for object clustering has achieved significant progresses recently, since the involvement of tactile modality can effectively improve clustering performance. However, the missing data (i.e., partial data) issues always happen due to occlusion and noises during the data collecting process. This issue is not well solved by most existing partial multi-view clustering methods for the heterogeneous modality challenge. Naively employing these methods would inevitably induce a negative effect and further hurt the performance. To solve the mentioned challenges, we propose …
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
Can faces acquired by low-cost depth sensors be useful to catch some characteristic details of the face? Typically the answer is no. However, new deep architectures can generate RGB images from data acquired in a different modality, such as depth data. In this paper, we propose a new \textit{Deterministic Conditional GAN}, trained on annotated RGB-D face datasets, effective for a face-to-face translation from depth to RGB. Although the network cannot reconstruct the exact somatic features for unknown individual faces, it is capable to reconstruct plausible faces; their appearance is accurate e…
Context-Aware Autoregressive Models for Multi-Conditional Image Generation
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）
Volumetric Conditional Score-based Residual Diffusion Model for PET/MR Denoising
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Neuroscience Research Institute（神经科学研究所）；Gachon University of Medicine and Science（嘉泉医科大学）
Conditional Diffusion Posterior Alignment for Sparse-View CT Reconstruction
Swiss Data Science Center (SDSC) in Paul Scherrer Institute (PSI)（瑞士数据科学中心（SDSC）在保罗·舍勒研究所（PSI））；Swiss Data Science Center (SDSC) and ETH Zurich（瑞士数据科学中心（SDSC）和苏黎世联邦理工学院）；Swiss Data Science Center (SDSC) and Paul Scherrer Institute (PSI)（瑞士数据科学中心（SDSC）和保罗·舍勒研究所（PSI））
Massachusetts General Hospital（马萨诸塞总医院）；Harvard Medical School（哈佛医学院）
Multimodal MRI provides complementary and clinically relevant information to probe tissue condition and to characterize various diseases. However, it is often difficult to acquire sufficiently many modalities from the same subject due to limitations in study plans, while quantitative analysis is still demanded. In this work, we propose a unified conditional disentanglement framework to synthesize any arbitrary modality from an input modality. Our framework hinges on a cycle-constrained conditional adversarial training approach, where it can extract a modality-invariant anatomical feature with …
Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching
Università Campus Bio-Medico di Roma（罗马 Campus Bio-Medico 大学）；Umeå University（于默奥大学）；UniCamillus – Saint Camillus International University of Health Sciences（UniCamillus – 圣卡米勒斯国际健康科学大学）
Tel Aviv University（特拉维夫大学）；Sheba Medical Center（谢巴医疗中心）；University College London Hospital（伦敦大学学院医院）
In this work we present a novel system for generation of virtual PET images using CT scans. We combine a fully convolutional network (FCN) with a conditional generative adversarial network (GAN) to generate simulated PET data from given input CT data. The synthesized PET can be used for false-positive reduction in lesion detection solutions. Clinically, such solutions may enable lesion detection and drug treatment evaluation in a CT-only environment, thus reducing the need for the more expensive and radioactive PET/CT scan. Our dataset includes 60 PET/CT scans from Sheba Medical center. We use…
Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation
ShanghaiTech University（上海科技大学）；Tencent PCG（腾讯平台与内容事业群）；Fudan University（复旦大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）；Shanghai Engineering Research Center of Energy Efficient and Custom AI IC（上海高效节能定制人工智能芯片工程研究中心）
Brain Tumour Removing and Missing Modality Generation using 3D WDM
Center Algoritmi / LASI, University of Minho（米尼奥大学Algoritmi中心 / LASI）；Computer Algorithms for Medicine Laboratory（计算机医学算法实验室）；Institute for AI in Medicine (IKIM), University Medicine Essen（埃森大学医学医院医学人工智能研究所）；Institute of Medical Informatics, University Hospital RWTH Aachen（亚琛工业大学医学院医学信息学研究所）；Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen（亚琛工业大学医学院口腔颌面外科）；Institute of Computer Graphics and Vision, Graz University of Technology（格拉茨工业大学计算机图形学与视觉研究所）；Cancer Research Center Cologne Essen (CCCE), University Medicine Essen（埃森大学医学医院科隆埃森癌症研究中心）；German Cancer Consortium (DKTK), Partner Site Essen（德国癌症联盟埃森合作站点）；Department of Physics, TU Dortmund University（多特蒙德工业大学物理系）
In this paper, we propose to use a Conditional Generative Adversarial Network (CGAN) for distilling (i.e. transferring) knowledge from sensor data and enhancing low-resolution target detection. In unconstrained surveillance settings, sensor measurements are often noisy, degraded, corrupted, and even missing/absent, thereby presenting a significant problem for multi-modal fusion. We therefore specifically tackle the problem of a missing modality in our attempt to propose an algorithm based on CGANs to generate representative information from the missing modalities when given some other availabl…
ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Northwestern Polytechnical University（西北工业大学）
Multi-modal medical image completion has been extensively applied to alleviate the missing modality issue in a wealth of multi-modal diagnostic tasks. However, for most existing synthesis methods, their inferences of missing modalities can collapse into a deterministic mapping from the available ones, ignoring the uncertainties inherent in the cross-modal relationships. Here, we propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) in modeling and stochastically sampling a target probability distribution, and…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Data from multi-modality provide complementary information in clinical prediction, but missing data in clinical cohorts limits the number of subjects in multi-modal learning context. Multi-modal missing imputation is challenging with existing methods when 1) the missing data span across heterogeneous modalities (e.g., image vs. non-image); or 2) one modality is largely missing. In this paper, we address imputation of missing data by modeling the joint distribution of multi-modal data. Motivated by partial bidirectional generative adversarial net (PBiGAN), we propose a new Conditional PBiGAN (C…
A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers
Centre des Matériaux (MAT)（材料中心）；CNRS（法国国家科学研究中心）；Centre de Mise en Forme des Matériaux (CEMEF)（材料成型中心）；Centre de Morphologie Mathématique (CMM)（数学形态学中心）；Mines Paris（巴黎高等矿业学院）；Université PSL（巴黎文理研究大学）
Learning modality invariant features is central to the problem of Visible-Thermal cross-modal Person Reidentification (VT-ReID), where query and gallery images come from different modalities. Existing works implicitly align the modalities in pixel and feature spaces by either using adversarial learning or carefully designing feature extraction modules that heavily rely on domain knowledge. We propose a simple but effective framework, MMD-ReID, that reduces the modality gap by an explicit discrepancy reduction constraint. MMD-ReID takes inspiration from Maximum Mean Discrepancy (MMD), a widely …
MS-DCANet: A Novel Segmentation Network For Multi-Modality COVID-19 Medical Images
College of Medical Informatics, Chongqing Medical University（重庆医科大学医学信息学院）；College of Intelligent Technology and Engineering, Chongqing University of Science and Technology（重庆科技学院智能技术与工程学院）
