Medical image translation (e.g. CT to MR) is a challenging task as it requires I) faithful translation of domain-invariant features (e.g. shape information of anatomical structures) and II) realistic synthesis of target-domain features (e.g. tissue appearance in MR). In this work, we propose Manifold Disentanglement Generative Adversarial Network (MDGAN), a novel image translation framework that explicitly models these two types of features. It employs a fully convolutional generator to model domain-invariant features, and it uses style codes to separately model target-domain features as a manifold. This design aims to explicitly disentangle domain-invariant features and domain-specific features while gaining individual control of both. The image translation process is formulated as a stylisation task, where the input is "stylised" (translated) into diverse target-domain images based on style codes sampled from the learnt manifold. We test MDGAN for multi-modal medical image translation, where we create two domain-specific manifold clusters on the manifold to translate segmentation maps into pseudo-CT and pseudo-MR images, respectively. We show that by traversing a path across the MR manifold cluster, the target output can be manipulated while still retaining the shape information from the input.
Decoupling Disease, Covariates, and Individual Variability: A Unified Disentanglement Framework for Medical Image Classification
Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Ruijin Hospital（瑞金医院）；Fudan University（复旦大学）；Shandong Provincial Hospital Affiliated to Shandong First Medical University（山东第一医科大学附属省立医院）；Air Force Medical University（空军军医大学）
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；Sensetime Research（商汤科技研究院）；Sichuan University（四川大学）；Sichuan Cancer Hospital and Institute（四川省肿瘤医院（四川省肿瘤研究所））
Generalization to previously unseen images with potential domain shifts and different styles is essential for clinically applicable medical image segmentation, and the ability to disentangle domain-specific and domain-invariant features is key for achieving Domain Generalization (DG). However, existing DG methods can hardly achieve effective disentanglement to get high generalizability. To deal with this problem, we propose an efficient Contrastive Domain Disentanglement and Style Augmentation (CDDSA) framework for generalizable medical image segmentation. First, a disentangle network is propo…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
Efficiently utilizing discriminative features is crucial for convolutional neural networks to achieve remarkable performance in medical image segmentation and is also important for model generalization across multiple domains, where letting model recognize domain-specific and domain-invariant information among multi-site datasets is a reasonable strategy for domain generalization. Unfortunately, most of the recent disentangle networks are not directly adaptable to unseen-domain datasets because of the limitations of offered data distribution. To tackle this deficiency, we propose Contrastive D…
Limited amount of labelled training data are a common problem in medical imaging. This makes it difficult to train a well-generalised model and therefore often leads to failure in unknown domains. Hippocampus segmentation from magnetic resonance imaging (MRI) scans is critical for the diagnosis and treatment of neuropsychatric disorders. Domain differences in contrast or shape can significantly affect segmentation. We address this issue by disentangling a T1-weighted MRI image into its content and domain. This separation enables us to perform a domain transfer and thus convert data from new so…
Fudan University（复旦大学）；University of Washington（华盛顿大学）
Multimodal groupwise registration aligns internal structures in a group of medical images. Current approaches to this problem involve developing similarity measures over the joint intensity profile of all images, which may be computationally prohibitive for large image groups and unstable under various conditions. To tackle these issues, we propose BInGo, a general unsupervised hierarchical Bayesian framework based on deep learning, to learn intrinsic structural representations to measure the similarity of multimodal images. Particularly, a variational auto-encoder with a novel posterior is pr…
Deep learning models exhibit limited generalizability across different domains. Specifically, transferring knowledge from available entangled domain features(source/target domain) and categorical features to new unseen categorical features in a target domain is an interesting and difficult problem that is rarely discussed in the current literature. This problem is essential for many real-world applications such as improving diagnostic classification or prediction in medical imaging. To address this problem, we propose Mutual-Information-based Disentangled Neural Networks (MIDNet) to extract ge…
Deep learning (DL) methods where interpretability is intrinsically considered as part of the model are required to better understand the relationship of clinical and imaging-based attributes with DL outcomes, thus facilitating their use in the reasoning behind medical decisions. Latent space representations built with variational autoencoders (VAE) do not ensure individual control of data attributes. Attribute-based methods enforcing attribute disentanglement have been proposed in the literature for classical computer vision tasks in benchmark data. In this paper, we propose a VAE approach, th…
Institute for AI in Medicine (IKIM), University Medicine Essen（埃森大学医学人工智能研究所（IKIM））；Computer Vision and Learning Lab (CVL), Heidelberg University（海德堡大学计算机视觉与学习实验室（CVL））
Deep neural networks are commonly used for medical purposes such as image generation, segmentation, or classification. Besides this, they are often criticized as black boxes as their decision process is often not human interpretable. Encouraging the latent representation of a generative model to be disentangled offers new perspectives of control and interpretability. Understanding the data generation process could help to create artificial medical data sets without violating patient privacy, synthesizing different data modalities, or discovering data generating characteristics. These character…
Imagia（Imagia公司）；Montréal Institute for Learning Algorithms (MILA)（蒙特利尔学习算法研究所）；Université de Montréal（蒙特利尔大学）；University of Victoria（维多利亚大学）；West China Hospital of Sichuan University（四川大学华西医院）
Synthetic medical image generation has a huge potential for improving healthcare through many applications, from data augmentation for training machine learning systems to preserving patient privacy. Conditional Adversarial Generative Networks (cGANs) use a conditioning factor to generate images and have shown great success in recent years. Intuitively, the information in an image can be divided into two parts: 1) content which is presented through the conditioning vector and 2) style which is the undiscovered information missing from the conditioning vector. Current practices in using cGANs f…
Disentanglement and Assessment of Shortcuts in Ophthalmological Retinal Imaging Exams
Faculdade de Engenharia, Universidade do Porto, Portugal（葡萄牙波尔图大学工程学院）；Institute for Systems and Computer Engineering, Technology and Science, Porto, Portugal（葡萄牙波尔图系统与计算机工程研究所）；University of Oxford, United Kingdom（英国牛津大学）；Federal University of São Paulo, São Paulo, Brasil（巴西圣保罗联邦大学）
Technical University of Munich（慕尼黑工业大学）；deepc GmbH（deepc有限公司）；Ludwig Maximilians University Munich（慕尼黑大学）；Kyung Hee University（庆熙大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Disentangled representations can be useful in many downstream tasks, help to make deep learning models more interpretable, and allow for control over features of synthetically generated images that can be useful in training other models that require a large number of labelled or unlabelled data. Recently, flow-based generative models have been proposed to generate realistic images by directly modeling the data distribution with invertible functions. In this work, we propose a new flow-based generative model framework, named GLOWin, that is end-to-end invertible and able to learn disentangled r…
University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）
Pseudo-healthy synthesis is the task of creating a subject-specific `healthy' image from a pathological one. Such images can be helpful in tasks such as anomaly detection and understanding changes induced by pathology and disease. In this paper, we present a model that is encouraged to disentangle the information of pathology from what seems to be healthy. We disentangle what appears to be healthy and where disease is as a segmentation map, which are then recombined by a network to reconstruct the input disease image. We train our models adversarially using either paired or unpaired settings, …
Disentanglement of Biological and Technical Factors via Latent Space Rotation in Clinical Imaging Improves Disease Pattern Discovery
Computational Imaging Research Lab, Department for Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门计算成像研究实验室）；Comprehensive Center for Artificial Intelligence in Medicine（医学人工智能综合中心）；Division of General and Pediatric Radiology, Department for Biomedical Imaging and Image-guided Therapy（普通放射科与儿科放射科，生物医学成像与影像引导治疗部门）；Medical University of Vienna（维也纳医学大学）
Causal Disentanglement for Robust Long-tail Medical Image Generation
School of Electrical and Information Engineering, Tianjin University, China（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento, Italy（特伦托大学信息工程与计算机科学系）；Fondazione Bruno Kessler, Italy（布鲁诺·克塞勒基金会）
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University（俄亥俄州立大学）；Imperial College London（伦敦帝国学院）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）
Motion-Guided Causal Disentanglement for Robust Multi-View Cine Cardiac MRI Diagnosis
University of Virginia（弗吉尼亚大学）；University of Chicago（芝加哥大学）；Ohio State University（俄亥俄州立大学）；St. Francis Hospital & Heart Center（圣弗朗西斯医院及心脏中心）；Hartford HealthCare（哈特福德医疗集团）；Instituto do Coração (InCor)（心脏研究所（InCor））；Atlantic Health System（大西洋健康系统）；Sociedad Italiana de Beneficencia (Hospital Italiano)（意大利慈善协会（意大利医院））
The success of deep learning has set new benchmarks for many medical image analysis tasks. However, deep models often fail to generalize in the presence of distribution shifts between training (source) data and test (target) data. One method commonly employed to counter distribution shifts is domain adaptation: using samples from the target domain to learn to account for shifted distributions. In this work we propose an unsupervised domain adaptation approach that uses graph neural networks and, disentangled semantic and domain invariant structural features, allowing for better performance acr…
Virginia Tech（弗吉尼亚理工大学）；University of Virginia（弗吉尼亚大学）
Advances in deep neural networks (DNNs) have shown tremendous promise in the medical domain. However, the deep learning tools that are helping the domain, can also be used against it. Given the prevalence of fraud in the healthcare domain, it is important to consider the adversarial use of DNNs in manipulating sensitive data that is crucial to patient healthcare. In this work, we present the design and implementation of a DNN-based image translation attack on biomedical imagery. More specifically, we propose Jekyll, a neural style transfer framework that takes as input a biomedical image of a …
University of Edinburgh（爱丁堡大学）；Canon Medical Research Europe Ltd.（佳能医疗欧洲研究有限公司）；The Alan Turing Institute（艾伦·图灵研究所）
Generalising deep models to new data from new centres (termed here domains) remains a challenge. This is largely attributed to shifts in data statistics (domain shifts) between source and unseen domains. Recently, gradient-based meta-learning approaches where the training data are split into meta-train and meta-test sets to simulate and handle the domain shifts during training have shown improved generalisation performance. However, the current fully supervised meta-learning approaches are not scalable for medical image segmentation, where large effort is required to create pixel-wise annotati…
Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，中国沈阳）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，中国沈阳）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，中国沈阳）；AiShiWeiLai AI Research, China（艾世维来人工智能研究，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学艾米人工智能研究所，加拿大埃德蒙顿，阿尔伯塔）
Beyond Euclidean Prototypes: Spectral Disentanglement and Geodesic Matching for Few-Shot Medical Image Segmentation
School of Microelectronics and Communication Engineering, Chongqing University（重庆大学微电子与通信工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）
Photoacoustic (PA) computed tomography (PACT) reconstructs the initial pressure distribution from raw PA signals. The standard reconstruction of medical image could cause the artifacts due to interferences or ill-posed setup. Recently, deep learning has been used to reconstruct the PA image with ill-posed conditions. Most works remove the artifacts from image domain, and compensate the limited-view from dataset. In this paper, we propose a jointed feature fusion framework (JEFF-Net) based on deep learning to reconstruct the PA image using limited-view data. The cross-domain features from limit…
Generative adversarial networks (GANs) have proven to be surprisingly efficient for image editing by inverting and manipulating the latent code corresponding to an input real image. This editing property emerges from the disentangled nature of the latent space. In this paper, we identify that the facial attribute disentanglement is not optimal, thus facial editing relying on linear attribute separation is flawed. We thus propose to improve semantic disentanglement with supervision. Our method consists in learning a proxy latent representation using normalizing flows, and we show that this lead…
Tel Aviv University（特拉维夫大学）；Technion - Israel Institute of Technology（以色列理工学院）
Training data is the key component in designing algorithms for medical image analysis and in many cases it is the main bottleneck in achieving good results. Recent progress in image generation has enabled the training of neural network based solutions using synthetic data. A key factor in the generation of new samples is controlling the important appearance features and potentially being able to generate a new sample of a specific class with different variants. In this work we suggest the synthesis of new data by mixing the class specified and unspecified representation of different factors in…
Northeastern University（东北大学）；University of Pittsburgh（匹兹堡大学）
The topic of generalizing machine learning models learned on a collection of source domains to unknown target domains is challenging. While many domain generalization (DG) methods have achieved promising results, they primarily rely on the source domains at train-time without manipulating the target domains at test-time. Thus, it is still possible that those methods can overfit to source domains and perform poorly on target domains. Driven by the observation that domains are strongly related to styles, we argue that reducing the gap between source and target styles can boost models' generaliza…
FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（西安光学精密机械研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
IMT School for Advanced Studies Lucca（IMT卢卡高级研究学院）；School of Engineering, University of Edinburgh（爱丁堡大学工程学院）
There has been an increasing focus in learning interpretable feature representations, particularly in applications such as medical image analysis that require explainability, whilst relying less on annotated data (since annotations can be tedious and costly). Here we build on recent innovations in style-content representations to learn anatomy, imaging characteristics (appearance) and temporal correlations. By introducing a self-supervised objective of predicting future cardiac phases we improve disentanglement. We propose a temporal transformer architecture that given an image conditioned on …
Helmholtz AI（亥姆霍兹AI）；Helmholtz Center Munich（慕尼黑亥姆霍兹中心）；Klinikum Rechts der Isar（右岸医院）；Technical University of Munich (TUM)（慕尼黑工业大学）
In recent years, data-driven machine learning (ML) methods have revolutionized the computer vision community by providing novel efficient solutions to many unsolved (medical) image analysis problems. However, due to the increasing privacy concerns and data fragmentation on many different sites, existing medical data are not fully utilized, thus limiting the potential of ML. Federated learning (FL) enables multiple parties to collaboratively train a ML model without exchanging local data. However, data heterogeneity (non-IID) among the distributed clients is yet a challenge. To this end, we pro…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；National Institute on Aging, National Institute of Health（美国国立卫生研究院国家老龄化研究所）
Disentangling anatomical and contrast information from medical images has gained attention recently, demonstrating benefits for various image analysis tasks. Current methods learn disentangled representations using either paired multi-modal images with the same underlying anatomy or auxiliary labels (e.g., manual delineations) to provide inductive bias for disentanglement. However, these requirements could significantly increase the time and cost in data collection and limit the applicability of these methods when such data are not available. Moreover, these methods generally do not guarantee …
Causal Disentanglement-Inspired Degradation Representation Learning for Full-Reference Image Quality Assessment
School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算机与人工智能学院，西南交通大学）；School of Transportation and Logistics, Southwest Jiaotong University（交通运输与物流学院，西南交通大学）；School of Physics, Northeast Normal University（物理学院，东北师范大学）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（计算机与人工智能学院，江西财经大学）；School of Information Management, Jiangxi University of Finance and Economics（信息管理学院，江西财经大学）
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（伊萨尔河畔临床教学中心）
Deep learning models used in medical image analysis are prone to raising reliability concerns due to their black-box nature. To shed light on these black-box models, previous works predominantly focus on identifying the contribution of input features to the diagnosis, i.e., feature attribution. In this work, we explore counterfactual explanations to identify what patterns the models rely on for diagnosis. Specifically, we investigate the effect of changing features within chest X-rays on the classifier's output to understand its decision mechanism. We leverage a StyleGAN-based approach (StyleE…
The Chinese University of Hong Kong（香港中文大学）；Imperial College Longdon（伦敦帝国理工学院）；Imsight Medical Technology Co. Ltd.（艾迈医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）
Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the s…
Seoul National University（首尔国立大学）；Soongsil University（崇实大学）
Individual tooth segmentation and identification from cone-beam computed tomography images are preoperative prerequisites for orthodontic treatments. Instance segmentation methods using convolutional neural networks have demonstrated ground-breaking results on individual tooth segmentation tasks, and are used in various medical imaging applications. While point-based detection networks achieve superior results on dental images, it is still a challenging task to distinguish adjacent teeth because of their similar topologies and proximate nature. In this study, we propose a point-based tooth loc…
Multimodal Medical Endoscopic Image Analysis via Progressive Disentangle-aware Contrastive Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；First Affiliated Hospital, Sun Yat-sen University（中山大学第一附属医院）；College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）
Variational Auto-Encoders (VAEs) have shown great potential in the unsupervised learning of data distributions. An VAE trained on normal images is expected to only be able to reconstruct normal images, allowing the localization of anomalous pixels in an image via manipulating information within the VAE ELBO loss. The ELBO consists of KL divergence loss (image-wise) and reconstruction loss (pixel-wise). It is natural and straightforward to use the later as the predictor. However, usually local anomaly added to a normal image can deteriorate the whole reconstructed image, causing segmentation us…
University of Copenhagen（哥本哈根大学）；Rigshospitalet（哥本哈根大学国立医院）
Medical imaging plays a vital role in modern diagnostics and treatment. The temporal nature of disease or treatment progression often results in longitudinal data. Due to the cost and potential harm, acquiring large medical datasets necessary for deep learning can be difficult. Medical image synthesis could help mitigate this problem. However, until now, the availability of GANs capable of synthesizing longitudinal volumetric data has been limited. To address this, we use the recent advances in latent space-based image editing to propose a novel joint learning scheme to explicitly embed tempor…
Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration
School of Data Science, Fudan University（复旦大学数据科学学院）；Department of Electrical and Electronic Engineering and I-X, Imperial College London（帝国理工学院电子与电气工程系）；Department of Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Department of Radiology and Imaging Sciences, University of Utah（犹他大学放射科与成像科学系）；Department of Radiology, University of Washington（华盛顿大学放射科）；Institute of Science and Technology for Brain-Inspired Intellengence, Fudan University（复旦大学脑启发智能科学技术研究院）
Multi-Domain Brain Vessel Segmentation Through Feature Disentanglement
Data Science Department, EURECOM（EURECOM数据科学部门）；Department of Medicine, Surgery and Neuroscience, University of Siena（锡耶纳大学医学、外科与神经科学系）；e-Health Center, Universitat Oberta de Catalunya（加泰罗尼亚开放大学e-健康中心）；Centre for Medical Image Computing (CMIC), Department of Medical Physics and Biomedical Engineering, University College London（伦敦大学学院医学影像计算中心）；Queen Square MS Centre, Department of Neuroinflammation, UCL Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London（伦敦大学学院Queen Square多发性硬化中心）；Sorbonne Université, Paris Brain Institute, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（巴黎大学巴黎脑研究所）；School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与成像科学学院）
Sichuan University（四川大学）；Chengdu University of Information Technology（成都信息工程大学）
Purpose: Different Magnetic resonance imaging (MRI) modalities of the same anatomical structure are required to present different pathological information from the physical level for diagnostic needs. However, it is often difficult to obtain full-sequence MRI images of patients owing to limitations such as time consumption and high cost. The purpose of this work is to develop an algorithm for target MRI sequences prediction with high accuracy, and provide more information for clinical diagnosis. Methods: We propose a deep learning based multi-modal computing model for MRI synthesis with featur…
Monash University（莫纳什大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61）；University of Melbourne（墨尔本大学）
Anomaly detection aims to recognize samples with anomalous and unusual patterns with respect to a set of normal data. This is significant for numerous domain applications, such as industrial inspection, medical imaging, and security enforcement. There are two key research challenges associated with existing anomaly detection approaches: (1) many approaches perform well on low-dimensional problems however the performance on high-dimensional instances, such as images, is limited; (2) many approaches often rely on traditional supervised approaches and manual engineering of features, while the top…
Enhancing Cross-Modal Medical Image Segmentation through Compositionality
University of Amsterdam（阿姆斯特丹大学）；The Netherlands Cancer Institute（荷兰癌症研究所）；Utrecht University（乌得勒支大学）；Maastricht University Medical Center（马斯特里赫特大学医学中心）
本文提出一种端到端跨模态医学图像分割框架，通过引入可学习的 von Mises-Fisher 核实现组合性归纳偏置与内容-风格解耦，有效提升了分割性能、可解释性并降低计算成本。
