Universal Topology Refinement for Medical Image Segmentation with Polynomial Feature Synthesis
Imperial College London（帝国理工学院）；University of Oxford（牛津大学）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学右侧医院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Liu Li, Hanchun Wang, Matthew Baugh, Qiang Ma, Weitong Zhang, Cheng Ouyang, Daniel Rueckert, Bernhard Kainz
备注：
Accepted by the 27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024)
Although existing medical image segmentation methods provide impressive pixel-wise accuracy, they often neglect topological correctness, making their segmentations unusable for many downstream tasks. One option is to retrain such models whilst including a topology-driven loss component. However, this is computationally expensive and often impractical. A better solution would be to have a versatile plug-and-play topology refinement method that is compatible with any domain-specific segmentation pipeline. Directly training a post-processing model to mitigate topological errors often fails as such models tend to be biased towards the topological errors of a target segmentation network. The diversity of these errors is confined to the information provided by a labelled training set, which is especially problematic for small datasets. Our method solves this problem by training a model-agnostic topology refinement network with synthetic segmentations that cover a wide variety of topological errors. Inspired by the Stone-Weierstrass theorem, we synthesize topology-perturbation masks with randomly sampled coefficients of orthogonal polynomial bases, which ensures a complete and unbiased representation. Practically, we verified the efficiency and effectiveness of our methods as being compatible with multiple families of polynomial bases, and show evidence that our universal plug-and-play topology refinement network outperforms both existing topology-driven learning-based and post-processing methods. We also show that combining our method with learning-based models provides an effortless add-on, which can further improve the performance of existing approaches.
Deep learning techniques have successfully been employed in numerous computer vision tasks including image segmentation. The techniques have also been applied to medical image segmentation, one of the most critical tasks in computer-aided diagnosis. Compared with natural images, the medical image is a gray-scale image with low-contrast (even with some invisible parts). Because some organs have similar intensity and texture with neighboring organs, there is usually a need to refine automatic segmentation results. In this paper, we propose an interactive deep refinement framework to improve the …
Peking University（北京大学）；Deepwise AI Lab（深睿医疗人工智能实验室）；Microsoft Research Asia（微软亚洲研究院）；The University of Hong Kong（香港大学）
As an important upstream task for many medical applications, supervised landmark localization still requires non-negligible annotation costs to achieve desirable performance. Besides, due to cumbersome collection procedures, the limited size of medical landmark datasets impacts the effectiveness of large-scale self-supervised pre-training methods. To address these challenges, we propose a two-stage framework for one-shot medical landmark localization, which first infers landmarks by unsupervised registration from the labeled exemplar to unlabeled targets, and then utilizes these noisy pseudo l…
Image segmentation is often performed on medical images for identifying diseases in clinical evaluation. Hence it has become one of the major research areas. Conventional image segmentation techniques are unable to provide satisfactory segmentation results for medical images as they contain irregularities. They need to be pre-processed before segmentation. In order to obtain the most suitable method for medical image segmentation, we propose a two stage algorithm. The first stage automatically generates a binary marker image of the region of interest using mathematical morphology. This marker …
Medical Image Segmentation via Single-Source Domain Generalization with Random Amplitude Spectrum Synthesis
School of Mechanical, Electrical & Information Engineering, Shandong University（山东大学机电与信息工程学院）；Shenzhen Research Institute of Shandong University（山东大学深圳研究院）；School of Computer Science and Technology, Shandong University of Finance and Economics（山东财经大学计算机科学与技术学院）；Shandong Provincial Key Laboratory of Digital Media Technology（山东省数字媒体技术重点实验室）
Towards Robust In-Context Learning for Medical Image Segmentation via Data Synthesis
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；University Hospital Tübingen（图宾根大学医院）；German Center for Mental Health（德国心理健康中心）
Medical image annotation typically requires expert knowledge and hence incurs time-consuming and expensive data annotation costs. To alleviate this burden, we propose a novel learning scenario, Exemplar Learning (EL), to explore automated learning processes for medical image segmentation with a single annotated image example. This innovative learning task is particularly suitable for medical image segmentation, where all categories of organs can be presented in one single image and annotated all at once. To address this challenging EL task, we propose an Exemplar Learning-based Synthesis Net (…
Multi-atlas segmentation (MAS) is a popular image segmentation technique for medical images. In this work, we improve the performance of MAS by correcting registration errors before label fusion. Specifically, we use a volumetric displacement field to refine registrations based on image anatomical appearance and predicted labels. We show the influence of the initial spatial alignment as well as the beneficial effect of using label information for MAS performance. Experiments demonstrate that the proposed refinement approach improves MAS performance on a 3D magnetic resonance dataset of the kne…
Explainable Continuous-Time Mask Refinement with Local Self-Similarity Priors for Medical Image Segmentation
School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India（计算机工程学院，印度克里希纳加尔工业技术学院，布巴内斯瓦尔，印度）；AmygdalaAI-India Lab, Bhubaneswar, India（AmygdalaAI-印度实验室，布巴内斯瓦尔，印度）
Southern University of Science and Technology（南方科技大学）；Technical University of Munich（慕尼黑工业大学）；University of Zurich（苏黎世大学）
Medical image synthesis has attracted increasing attention because it could generate missing image data, improving diagnosis and benefits many downstream tasks. However, so far the developed synthesis model is not adaptive to unseen data distribution that presents domain shift, limiting its applicability in clinical routine. This work focuses on exploring domain adaptation (DA) of 3D image-to-image synthesis models. First, we highlight the technical difference in DA between classification, segmentation and synthesis models. Second, we present a novel efficient adaptation approach based on 2D v…
Deep convolutional neural networks for image segmentation do not learn the label structure explicitly and may produce segmentations with an incorrect structure, e.g., with disconnected cylindrical structures in the segmentation of tree-like structures such as airways or blood vessels. In this paper, we propose a novel label refinement method to correct such errors from an initial segmentation, implicitly incorporating information about label structure. This method features two novel parts: 1) a model that generates synthetic structural errors, and 2) a label appearance simulation network that …
Bilkent University（比尔肯特大学）；National Magnetic Resonance Research Center (UMRAM)（国家磁共振研究中心（UMRAM））；Sabuncu Brain Research Center（萨本库脑研究中心）
Generative adversarial models with convolutional neural network (CNN) backbones have recently been established as state-of-the-art in numerous medical image synthesis tasks. However, CNNs are designed to perform local processing with compact filters, and this inductive bias compromises learning of contextual features. Here, we propose a novel generative adversarial approach for medical image synthesis, ResViT, that leverages the contextual sensitivity of vision transformers along with the precision of convolution operators and realism of adversarial learning.} ResViT's generator employs a cent…
University of Chinese Academy of Sciences（中国科学院大学）；Inception Institute of Artificial Intelligence（启元人工智能研究院）；UBTech Research（优必选研究院）；ShanghaiTech University（上海科技大学）；Southern University of Science and Technology（南方科技大学）
Generative Adversarial Networks (GANs) have the capability of synthesizing images, which have been successfully applied to medical image synthesis tasks. However, most of existing methods merely consider the global contextual information and ignore the fine foreground structures, e.g., vessel, skeleton, which may contain diagnostic indicators for medical image analysis. Inspired by human painting procedure, which is composed of stroking and color rendering steps, we propose a Sketching-rendering Unconditional Generative Adversarial Network (SkrGAN) to introduce a sketch prior constraint to gui…
Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）；Nanyang Technological University（南洋理工大学）
Xidian University（西安电子科技大学）；Northwestern Polytechnical University（西北工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Nankai University（南开大学）；City University of Hong Kong（香港城市大学）
Due to the constraints of the imaging device and high cost in operation time, computer tomography (CT) scans are usually acquired with low intra-slice resolution. Improving the intra-slice resolution is beneficial to the disease diagnosis for both human experts and computer-aided systems. To this end, this paper builds a novel medical slice synthesis to increase the between-slice resolution. Considering that the ground-truth intermediate medical slices are always absent in clinical practice, we introduce the incremental cross-view mutual distillation strategy to accomplish this task in the sel…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai Jiao Tong University（上海交通大学）
Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and risk, the acquisition of certain image modalities could be limited. To address this issue, many cross-modality medical image synthesis methods have been proposed. However, the current methods cannot well model the hard-to-synthesis regions (e.g., tumor or lesion regions). To address this issue, we propose a simple but effective strategy, that is, we propose a dual-discriminator (dual-D) adversarial learning system, in which, a global-D is used to make an overall eval…
HAGAN: Hybrid Augmented Generative Adversarial Network for Medical Image Synthesis
Beijing University of Posts and Telecommunications（北京邮电大学）；Dalian University of Technology（大连理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis
Heidelberg University Hospital（海德堡大学医院）；German Centre for Cardiovascular Research（德国心血管研究中心）；Heidelberg University（海德堡大学）；University Medical Center Mannheim（曼海姆大学医学中心）
This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were summarized in the discussion part.
Deep learning-based single image super-resolution (SISR) methods face various challenges when applied to 3D medical volumetric data (i.e., CT and MR images) due to the high memory cost and anisotropic resolution, which adversely affect their performance. Furthermore, mainstream SISR methods are designed to work over specific upsampling factors, which makes them ineffective in clinical practice. In this paper, we introduce a Spatially Aware Interpolation NeTwork (SAINT) for medical slice synthesis to alleviate the memory constraint that volumetric data poses. Compared to other super-resolution …
As a powerful technique in medical imaging, image synthesis is widely used in applications such as denoising, super resolution and modality transformation etc. Recently, the revival of deep neural networks made immense progress in the field of medical imaging. Although many deep leaning based models have been proposed to improve the image synthesis accuracy, the evaluation of the model uncertainty, which is highly important for medical applications, has been a missing part. In this work, we propose to use Bayesian conditional generative adversarial network (GAN) with concrete dropout to improv…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）；Univ. Lille（里尔大学）；Inserm（法国国家健康与医学研究院）；CHU Lille（里尔大学医院）
Image synthesis via Generative Adversarial Networks (GANs) of three-dimensional (3D) medical images has great potential that can be extended to many medical applications, such as, image enhancement and disease progression modeling. However, current GAN technologies for 3D medical image synthesis need to be significantly improved to be readily adapted to real-world medical problems. In this paper, we extend the state-of-the-art StyleGAN2 model, which natively works with two-dimensional images, to enable 3D image synthesis. In addition to the image synthesis, we investigate the controllability a…
On Differentially Private 3D Medical Image Synthesis with Controllable Latent Diffusion Models
Technical University of Munich（慕尼黑工业大学）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Universitat de Barcelona（巴塞罗那大学）；Imperial College London（帝国理工学院）；King’s College London（伦敦国王学院）；Helmholtz AI（亥姆霍兹人工智能研究所）
本文提出基于医学属性条件的 Latent Diffusion Models，通过差分隐私训练生成 3D 心脏 MRI 图像，并首次在 3D 医学图像生成中量化差分隐私，发现预训练显著提升图像质量。
I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling
Department of Electrical and Electronics Engineering, and National Magnetic Resonance Research Center, Bilkent University（电子工程系和国家磁共振研究中心，比尔肯特大学）；Department of Electrical Engineering, Stanford University（电气工程系，斯坦福大学）
Exploiting learning algorithms under scarce data regimes is a limitation and a reality of the medical imaging field. In an attempt to mitigate the problem, we propose a data augmentation protocol based on generative adversarial networks. We condition the networks at a pixel-level (segmentation mask) and at a global-level information (acquisition environment or lesion type). Such conditioning provides immediate access to the image-label pairs while controlling global class specific appearance of the synthesized images. To stimulate synthesis of the features relevant for the segmentation task, a…
University of Bourgogne Franche-Comte（勃艮第-弗朗什孔泰大学）；CASIS inc.（CASIS公司）；University of Sherbrooke（舍布鲁克大学）；University Hospital of Dijon（第戎大学医院）
Generative Adversarial Networks (GANs) have become increasingly powerful, generating mind-blowing photorealistic images that mimic the content of datasets they were trained to replicate. One recurrent theme in medical imaging is whether GANs can also be effective at generating workable medical data as they are for generating realistic RGB images. In this paper, we perform a multi-GAN and multi-application study to gauge the benefits of GANs in medical imaging. We tested various GAN architectures from basic DCGAN to more sophisticated style-based GANs on three medical imaging modalities and org…
As an effective way to integrate the information contained in multiple medical images under different modalities, medical image synthesis and fusion have emerged in various clinical applications such as disease diagnosis and treatment planning. In this paper, an invertible and variable augmented network (iVAN) is proposed for medical image synthesis and fusion. In iVAN, the channel number of the network input and output is the same through variable augmentation technology, and data relevance is enhanced, which is conducive to the generation of characterization information. Meanwhile, the inver…
Recently, the cycle-consistent generative adversarial networks (CycleGAN) has been widely used for synthesis of multi-domain medical images. The domain-specific nonlinear deformations captured by CycleGAN make the synthesized images difficult to be used for some applications, for example, generating pseudo-CT for PET-MR attenuation correction. This paper presents a deformation-invariant CycleGAN (DicycleGAN) method using deformable convolutional layers and new cycle-consistency losses. Its robustness dealing with data that suffer from domain-specific nonlinear deformations has been evaluated t…
Deformation-aware GAN for Medical Image Synthesis with Substantially Misaligned Pairs
Australian e-Health Research Centre, CSIRO（澳大利亚电子健康研究中心（联邦科学与工业研究组织））；Liverpool and Macarthur Cancer Therapy Centres and Ingham Institute（利物浦与麦卡瑟癌症治疗中心及英厄姆研究所）；Child Health Research Centre, University of Queensland（昆士兰大学儿童健康研究中心）；Queensland Children’s Hospital（昆士兰儿童医院）；Data61, CSIRO（Data61（联邦科学与工业研究组织下属机构））；Siemens Healthcare Pty Ltd（西门子医疗私人有限公司）；Siemens Healthineers AG（西门子医疗保健股份公司）
Pseudo-normality synthesis, which computationally generates a pseudo-normal image from an abnormal one (e.g., with lesions), is critical in many perspectives, from lesion detection, data augmentation to clinical surgery suggestion. However, it is challenging to generate high-quality pseudo-normal images in the absence of the lesion information. Thus, expensive lesion segmentation data have been introduced to provide lesion information for the generative models and improve the quality of the synthetic images. In this paper, we aim to alleviate the need of a large amount of lesion segmentation d…
Temporal Evolution of Knee Osteoarthritis: A Diffusion-based Morphing Model for X-ray Medical Image Synthesis
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Orleans（奥尔良大学）；IDP Institute（IDP研究所）；Jiangsu Institute of Hematology（江苏省血液研究所）；First Affiliated Hospital of Soochow University（苏州大学附属第一医院）；Cancer Hospital of Pernambuco（伯南布哥癌症医院）
StructDiff: Structure-aware Diffusion Model for 3D Fine-grained Medical Image Synthesis
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及跨学科应用关键实验室（东南大学），中华人民共和国教育部，中国）；Jiangsu Province Joint International Research Laboratory of Medical Information Processing, Southeast University, Nanjing, China（江苏省医疗信息处理联合国际研究实验室（东南大学），南京市，中国）；Univ Rennes, CHU Rennes, Inserm, LTSI– UMR 1099, F-35000 Rennes, France（里昂大学，里昂大学医院，Inserm，LTSI– UMR 1099，法国里昂，法国）；Shanghai AI Laboratory（上海人工智能实验室）；School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China（计算机科学与技术学院，北京理工大学，北京，中国）；Department of Radiology, Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, China（放射科，南京大学附属医院，南京大学，南京，中国）
Universitat de Barcelona（巴塞罗那大学）；University of Arkansas for Medical Sciences（阿肯色医科大学）
Synthetic data generated by generative models can enhance the performance and capabilities of data-hungry deep learning models in medical imaging. However, there is (1) limited availability of (synthetic) datasets and (2) generative models are complex to train, which hinders their adoption in research and clinical applications. To reduce this entry barrier, we propose medigan, a one-stop shop for pretrained generative models implemented as an open-source framework-agnostic Python library. medigan allows researchers and developers to create, increase, and domain-adapt their training data in jus…
Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）；Zhejiang University（浙江大学）
Reducing the radiation dose in computed tomography (CT) is important to mitigate radiation-induced risks. One option is to employ a well-trained model to compensate for incomplete information and map sparse-view measurements to the CT reconstruction. However, reconstruction from sparsely sampled measurements is insufficient to uniquely characterize an object in CT, and a learned prior model may be inadequate for unencountered cases. Medical modal translation from magnetic resonance imaging (MRI) to CT is an alternative but may introduce incorrect information into the synthesized CT images in a…
Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Harvard Medical School and the Brigham and Women’s Hospital（哈佛医学院和布里洛妇产科医院）；MIND Team, Inria Saclay, Université Paris-Saclay, Palaiseau, France and Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（MIND团队、Inria Saclay、巴黎萨克雷大学、Palaiseau，法国以及索邦大学、巴黎脑研究所-ICM、CNRS、Inria、Inserm、AP-HP、皮蒂埃萨勒普蒂耶医院）；Massachusetts Institute of Technology（麻省理工学院）
