AI-driven tumor analysis has garnered increasing attention in healthcare. However, its progress is significantly hindered by the lack of annotated tumor cases, which requires radiologists to invest a lot of effort in collecting and annotation. In this paper, we introduce a highly practical solution for robust tumor synthesis and segmentation, termed FreeTumor, which refers to annotation-free synthetic tumors and our desire to free patients that suffering from tumors. Instead of pursuing sophisticated technical synthesis modules, we aim to design a simple yet effective tumor synthesis paradigm to unleash the power of large-scale data. Specifically, FreeTumor advances existing methods mainly from three aspects: (1) Existing methods only leverage small-scale labeled data for synthesis training, which limits their ability to generalize well on unseen data from different sources. To this end, we introduce the adversarial training strategy to leverage large-scale and diversified unlabeled data in synthesis training, significantly improving tumor synthesis. (2) Existing methods largely ignored the negative impact of low-quality synthetic tumors in segmentation training. Thus, we employ an adversarial-based discriminator to automatically filter out the low-quality synthetic tumors, which effectively alleviates their negative impact. (3) Existing methods only used hundreds of cases in tumor segmentation. In FreeTumor, we investigate the data scaling law in tumor segmentation by scaling up the dataset to 11k cases. Extensive experiments demonstrate the superiority of FreeTumor, e.g., on three tumor segmentation benchmarks, average $+8.9\%$ DSC over the baseline that only using real tumors and $+6.6\%$ DSC over the state-of-the-art tumor synthesis method. Code will be available.
FreeTumor: Large-Scale Generative Tumor Synthesis in Computed Tomography Images for Improving Tumor Recognition
The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯AI实验室）；Harvard University（哈佛大学）；The Chinese University of Hong Kong（香港中文大学）；The Third Affiliated Hospital of Southern Medical University（南方医科大学第三附属医院）；Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University（中山大学孙逸仙纪念医院）；The University of Hong Kong（香港大学）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港产学研协同创新研究院）
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea（延世大学电气与计算机工程系）；Department of Radiology and Center for Imaging Science, Samsung Medical Center, Sungkyunkwan University School of Medicine, Suwon, Korea（延世大学医学院放射科与成像科学中心）
TumorFlow: Physics-Guided Longitudinal MRI Synthesis of Glioblastoma Growth
Technical University of Munich (TUM), Germany（慕尼黑技术大学（TUM））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；AI for Image-Guided Diagnosis and Therapy（图像引导诊断与治疗的AI）；Chair for AI in Healthcare and Medicine, TUM and TUM University Hospital, Munich, Germany（医疗与医学中的AI主任，慕尼黑技术大学及慕尼黑大学医院）
University of Chinese Academy of Sciences（中国科学院大学）；Emory University（埃默里大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Southern California（南加利福尼亚大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Tianjin University（天津大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；La Trobe University（乐卓博大学）；Tianjin University of Traditional Chinese Medicine（天津中医药大学）
The insufficiency of annotated medical imaging scans for cancer makes it challenging to train and validate data-hungry deep learning models in precision oncology. We propose a new richer generative adversarial network for free-form 3D tumor/lesion synthesis in computed tomography (CT) images. The network is composed of a new richer convolutional feature enhanced dilated-gated generator (RicherDG) and a hybrid loss function. The RicherDG has dilated-gated convolution layers to enable tumor-painting and to enlarge perceptive fields; and it has a novel richer convolutional feature association bra…
Diffusion-Based Tumor Inpainting for Renal Segmentation under Clinical Data Scarcity
University of Tartu（塔尔图大学）；University of Bern（伯尔尼大学）；McGill University（麦吉尔大学）；Mila – Québec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；Better Medicine OÜ（Better Medicine 公司）；STACC OÜ（STACC 公司）
Johns Hopkins University（约翰斯·霍普金斯大学）；Shenzhen Technology University（深圳技术大学）；Sun Yat-sen University（中山大学）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；Hong Kong Polytechnic University（香港理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；NVIDIA（英伟达公司）；University of Bologna（博洛尼亚大学）；Italian Institute of Technology（意大利技术研究院）；University of California, San Francisco（加利福尼亚大学旧金山分校）
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Multi-modal magnetic resonance (MR) imaging provides great potential for diagnosing and analyzing brain gliomas. In clinical scenarios, common MR sequences such as T1, T2 and FLAIR can be obtained simultaneously in a single scanning process. However, acquiring contrast enhanced modalities such as T1ce requires additional time, cost, and injection of contrast agent. As such, it is clinically meaningful to develop a method to synthesize unavailable modalities which can also be used as additional inputs to downstream tasks (e.g., brain tumor segmentation) for performance enhancing. In this work, …
Magnetic Resonance (MR) images of different modalities can provide complementary information for clinical diagnosis, but whole modalities are often costly to access. Most existing methods only focus on synthesizing missing images between two modalities, which limits their robustness and efficiency when multiple modalities are missing. To address this problem, we propose a multi-modality generative adversarial network (MGAN) to synthesize three high-quality MR modalities (FLAIR, T1 and T1ce) from one MR modality T2 simultaneously. The experimental results show that the quality of the synthesize…
Korea Advanced Institute of Science and Technology (KAIST)（韩国高等科学技术研究院（KAIST））
Medical image analysis using deep neural networks has been actively studied. Deep neural networks are trained by learning data. For accurate training of deep neural networks, the learning data should be sufficient, of good quality, and should have a generalized property. However, in medical images, it is difficult to acquire sufficient patient data because of the difficulty of patient recruitment, the burden of annotation of lesions by experts, and the invasion of patients' privacy. In comparison, the medical images of healthy volunteers can be easily acquired. Using healthy brain images, the …
Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China（电子与电气工程系，南方科技大学，深圳，中国）；Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China（电气与电子工程系，香港大学，香港特别行政区，中国）；Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China（放射科，中大医院，医学院，东南大学，南京，中国）；Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, China（嘉兴研究所，南方科技大学，嘉兴，中国）
An Ensemble Approach for Brain Tumor Segmentation and Synthesis
Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗G.艾伦计算机科学学院）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Department of Bioengineering, University of Washington（华盛顿大学生物工程系）；Department of Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Department of Mechanical Engineering, University of Washington（华盛顿大学机械工程系）；Watauga High School（沃托加高中）；Department of Neurological Surgery, University of Washington（华盛顿大学神经外科系）
Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities
University of Science and Technology of China（中国科学技术大学）；Tianjin University（天津大学）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequences
Interdisciplinary Program in Cancer Biology, Seoul National University College of Medicine（首尔大学医学院癌症生物学跨学科项目）；Department of Radiology, Seoul National University Hospital（首尔大学医院放射科）；Department of Radiology, Seoul National University College of Medicine（首尔大学医学院放射科）；Healthcare AI Research Institute, Seoul National University Hospital（首尔大学医院医疗人工智能研究所）
CAVM: Conditional Autoregressive Vision Model for Contrast-Enhanced Brain Tumor MRI Synthesis
School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）；School of Integrated Circuits and Electronics, Beijing Institute of Technology（北京理工大学集成电路与电子学院）
The Brain Tumor Segmentation (BraTS) Challenge 2023: Brain MR Image Synthesis for Tumor Segmentation (BraSyn)
University of Zurich（苏黎世大学）；Technical University Munich（慕尼黑工业大学）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学右岸医院）；Mayo Clinic（梅奥诊所）；Helmholtz AI, Helmholtz Munich（亥姆霍兹慕尼黑亥姆霍兹人工智能研究所）；TranslaTUM - Central Institute for Translational Cancer Research, Technical University of Munich（慕尼黑工业大学转化肿瘤研究中心TranslaTUM）；Children’s National Hospital（全国儿童医院）；George Washington University（乔治华盛顿大学）；Finnish Center for Artificial Intelligence（芬兰人工智能中心）；Athinoula A. Martinos Center for Biomedical Imaging, Harvard Medical School（哈佛医学院阿西努拉·A·马蒂诺斯生物医学成像中心）；Crestview Radiology（克雷斯特维尤放射科）；University of Pennsylvania（宾夕法尼亚大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；Yale University（耶鲁大学）；Children’s Hospital of Philadelphia, University of Pennsylvania（宾夕法尼亚大学费城儿童医院）；Duke University Medical Center（杜克大学医学中心）
University of Science and Technology of China（中国科学技术大学）；University of Miami（迈阿密大学）；the First Affiliated Hospital of USTC（中国科学技术大学附属第一医院）
Automatic deep learning segmentation models has been shown to improve both the segmentation efficiency and the accuracy. However, training a robust segmentation model requires considerably large labeled training samples, which may be impractical. This study aimed to develop a deep learning framework for generating synthetic lesions that can be used to enhance network training. The lesion synthesis network is a modified generative adversarial network (GAN). Specifically, we innovated a partial convolution strategy to construct an Unet-like generator. The discriminator is designed using Wasserst…
Institute for Medical Imaging Technology, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200030, China（中国上海200030，上海交通大学生物医学工程学院医学影像技术研究所）；Department of Radiology and Biomedical Research Imaging Center (BRIC), University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA（美国北卡罗来纳州教堂山27599，北卡罗来纳大学教堂山分校放射学与生物医学研究成像中心（BRIC））；North China Electric Power University, Beijing, China（中国北京华北电力大学）；Shanghai United Imaging Intelligence Co., Ltd, Shanghai, China（中国上海上海联影智能科技有限公司）；Shanghai Cancer Center, Fudan University, Shanghai, China（中国上海复旦大学附属肿瘤医院）
Thermal ablation is a minimally invasive procedure for treat-ing small or unresectable tumors. Although CT is widely used for guiding ablation procedures, the contrast of tumors against surrounding normal tissues in CT images is often poor, aggravating the difficulty in accurate thermal ablation. In this paper, we propose a fast MR-CT image registration method to overlay a pre-procedural MR (pMR) image onto an intra-procedural CT (iCT) image for guiding the thermal ablation of liver tumors. By first using a Cycle-GAN model with mutual information constraint to generate synthesized CT (sCT) ima…
3DGAUnet: 3D generative adversarial networks with a 3D U-Net based generator to achieve the accurate and effective synthesis of clinical tumor image data for pancreatic cancer
University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；University of Nebraska Medical Center（内布拉斯加大学医学中心）；University of Rochester Medical Center（罗切斯特大学医学中心）
Exploiting learning algorithms under scarce data regimes is a limitation and a reality of the medical imaging field. In an attempt to mitigate the problem, we propose a data augmentation protocol based on generative adversarial networks. We condition the networks at a pixel-level (segmentation mask) and at a global-level information (acquisition environment or lesion type). Such conditioning provides immediate access to the image-label pairs while controlling global class specific appearance of the synthesized images. To stimulate synthesis of the features relevant for the segmentation task, a…
University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）
Positron emission tomography (PET) image synthesis plays an important role, which can be used to boost the training data for computer aided diagnosis systems. However, existing image synthesis methods have problems in synthesizing the low resolution PET images. To address these limitations, we propose multi-channel generative adversarial networks (M-GAN) based PET image synthesis method. Different to the existing methods which rely on using low-level features, the proposed M-GAN is capable to represent the features in a high-level of semantic based on the adversarial learning concept. In addit…
Medical School of Nanjing University（南京大学医学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学国家健康医疗大数据研究院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computer Science and Technology, Nanjing University of Science and Technology（南京理工大学计算机科学与技术学院）；Department of Radiology, Nanjing Drum Tower Hospital, Nanjing University Medical School（南京大学医学院附属鼓楼医院放射科）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Department of Research and Development, Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司研发部）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）
Accurate segmentation of brain tumors from multi-modal Magnetic Resonance (MR) images is essential in brain tumor diagnosis and treatment. However, due to the existence of domain shifts among different modalities, the performance of networks decreases dramatically when training on one modality and performing on another, e.g., train on T1 image while performing on T2 image, which is often required in clinical applications. This also prohibits a network from being trained on labeled data and then transferred to unlabeled data from a different domain. To overcome this, unsupervised domain adaptat…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai Jiao Tong University（上海交通大学）
Medical imaging plays a critical role in various clinical applications. However, due to multiple considerations such as cost and risk, the acquisition of certain image modalities could be limited. To address this issue, many cross-modality medical image synthesis methods have been proposed. However, the current methods cannot well model the hard-to-synthesis regions (e.g., tumor or lesion regions). To address this issue, we propose a simple but effective strategy, that is, we propose a dual-discriminator (dual-D) adversarial learning system, in which, a global-D is used to make an overall eval…
NVIDIA Corporation（英伟达公司）；MGH & BWH Center for Clinical Data Science（麻省总医院与布列根和妇女医院临床数据科学中心）；Mayo Clinic（妙佑医疗国际）
Data diversity is critical to success when training deep learning models. Medical imaging data sets are often imbalanced as pathologic findings are generally rare, which introduces significant challenges when training deep learning models. In this work, we propose a method to generate synthetic abnormal MRI images with brain tumors by training a generative adversarial network using two publicly available data sets of brain MRI. We demonstrate two unique benefits that the synthetic images provide. First, we illustrate improved performance on tumor segmentation by leveraging the synthetic images…
Shenzhen University（深圳大学）；Islamic Azad University（伊斯兰阿扎德大学）
Magnetic resonance imaging (MRI) provides varying tissue contrast images of internal organs based on a strong magnetic field. Despite the non-invasive advantage of MRI in frequent imaging, the low contrast MR images in the target area make tissue segmentation a challenging problem. This paper demonstrates the potential benefits of image-to-image translation techniques to generate synthetic high tissue contrast (HTC) images. Notably, we adopt a new cycle generative adversarial network (CycleGAN) with an attention mechanism to increase the contrast within underlying tissues. The attention block,…
INHA University（仁荷大学）；Kyung Hee University（庆熙大学）；Team Elysium Inc.（Team Elysium公司）；Pusan National University Hospital（釜山大学医院）
MR imaging will play a very important role in radiotherapy treatment planning for segmentation of tumor volumes and organs. However, the use of MR-based radiotherapy is limited because of the high cost and the increased use of metal implants such as cardiac pacemakers and artificial joints in aging society. To improve the accuracy of CT-based radiotherapy planning, we propose a synthetic approach that translates a CT image into an MR image using paired and unpaired training data. In contrast to the current synthetic methods for medical images, which depend on sparse pairwise-aligned data or pl…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
Contrast-enhanced T1 (T1ce) is one of the most essential magnetic resonance imaging (MRI) modalities for diagnosing and analyzing brain tumors, especially gliomas. In clinical practice, common MRI modalities such as T1, T2, and fluid attenuation inversion recovery are relatively easy to access while T1ce is more challenging considering the additional cost and potential risk of allergies to the contrast agent. Therefore, it is of great clinical necessity to develop a method to synthesize T1ce from other common modalities. Current paired image translation methods typically have the issue of requ…
The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting
Helmholtz AI, Germany（亥姆霍兹人工智能研究所）；Technical University of Munich, Germany（慕尼黑工业大学）；Helmholtz Munich, Germany（亥姆霍兹慕尼黑中心）；University of Zurich, Switzerland（苏黎世大学）；Klinikum Rechts der Isar, Munich, Germany（慕尼黑右岸医院）；SETLabs Research GmbH, Healthcare Technologies, Munich, Germany（SETLabs研究有限公司）；Ludwig Maximilian University, Munich, Germany（路德维希马克西米利安慕尼黑大学）
University of Ljubljana（卢布尔雅那大学）；XLAB d.o.o.（XLAB公司）
Anomaly detection in visual data refers to the problem of differentiating abnormal appearances from normal cases. Supervised approaches have been successfully applied to different domains, but require an abundance of labeled data. Due to the nature of how anomalies occur and their underlying generating processes, it is hard to characterize and label them. Recent advances in deep generative-based models have sparked interest in applying such methods for unsupervised anomaly detection and have shown promising results in medical and industrial inspection domains. In this work we evaluate a crucia…
University of Copenhagen（哥本哈根大学）；Rigshospitalet（哥本哈根大学国立医院）
Medical imaging plays a vital role in modern diagnostics and treatment. The temporal nature of disease or treatment progression often results in longitudinal data. Due to the cost and potential harm, acquiring large medical datasets necessary for deep learning can be difficult. Medical image synthesis could help mitigate this problem. However, until now, the availability of GANs capable of synthesizing longitudinal volumetric data has been limited. To address this, we use the recent advances in latent space-based image editing to propose a novel joint learning scheme to explicitly embed tempor…
Imagia Inc.（Imagia 公司）；Université de Sherbrooke（舍布鲁克大学）；Université de Montréal（蒙特利尔大学）
We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, …
The ability to detect failures and anomalies are fundamental requirements for building reliable systems for computer vision applications, especially safety-critical applications of semantic segmentation, such as autonomous driving and medical image analysis. In this paper, we systematically study failure and anomaly detection for semantic segmentation and propose a unified framework, consisting of two modules, to address these two related problems. The first module is an image synthesis module, which generates a synthesized image from a segmentation layout map, and the second is a comparison m…
VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation
Center Algoritmi / LASI, University of Minho（中心Algoritmi / LASI，明霍大学）；Institute for Artificial Intelligence in Medicine (IKIM)（医学人工智能研究所）；Essen University Hospital (AöR)（埃森大学医院（AöR））；University of Duisburg-Essen（杜伊斯堡-埃森大学）；Cancer Research Center Cologne Essen (CCCE)（科隆-埃森癌症研究中心（CCCE））；West German Cancer Center（西德癌症中心）；University Hospital Essen (AöR)（埃森大学医院（AöR））；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；Partner site University Hospital Essen (AöR)（埃森大学医院（AöR）合作站点）；Institute of Computer Graphics and Vision（计算机图形与视觉研究所）；Graz University of Technology（格拉茨技术大学）；Institute of Medical Informatics（医学信息学研究所）；University Hospital RWTH Aachen（亚琛RWTH大学医院）；Department of Oral and Maxillofacial Surgery（口腔和颌面外科部门）；TU Dortmund University（多特蒙德技术大学）；Center for Virtual and Extended Reality in Medicine (ZvRM)（医学虚拟和扩展现实中心（ZvRM））；Faculty of Computer Science（计算机科学学院）
