Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion
The Chinese University of Hong Kong（香港中文大学）；Imperial College Longdon（伦敦帝国理工学院）；Imsight Medical Technology Co. Ltd.（艾迈医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）
Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the segmentation task. With enhanced modality-invariance, the disentangled content code from each modality is fused into a shared representation which gains robustness to missing data. The fusion is achieved via a learning-based strategy to gate the contribution of different modalities at different locations. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset. With competitive performance to the state-of-the-art approaches for full modality, our method achieves outstanding robustness under various missing modality(ies) situations, significantly exceeding the state-of-the-art method by over 16% in average for Dice on whole tumor segmentation.
For Domain Generalizable Object Detection (DGOD), Disentangled Representation Learning (DRL) helps a lot by explicitly disentangling Domain-Invariant Representations (DIR) from Domain-Specific Representations (DSR). Considering the domain category is an attribute of input data, it should be feasible for networks to fit a specific mapping which projects DSR into feature channels exclusive to domain-specific information, and thus much cleaner disentanglement of DIR from DSR can be achieved simply on channel dimension. Inspired by this idea, we propose a novel DRL method for DGOD, which is termed…
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
Due to the difficulties of obtaining multimodal paired images in clinical practice, recent studies propose to train brain tumor segmentation models with unpaired images and capture complementary information through modality translation. However, these models cannot fully exploit the complementary information from different modalities. In this work, we thus present a novel two-step (intra-modality and inter-modality) curriculum disentanglement learning framework to effectively utilize privileged semi-paired images, i.e. limited paired images that are only available in training, for brain tumor …
CausalDisenSeg: A Causality-Guided Disentanglement Framework with Counterfactual Reasoning for Robust Brain Tumor Segmentation Under Missing Modalities
School of Information Engineering, Nanchang University（南昌大学信息工程学院）；School of Mathematics and Computer Sciences, Nanchang University（南昌大学数学与计算机科学学院）
Variational AutoEncoders (VAEs) provide a means to generate representational latent embeddings. Previous research has highlighted the benefits of achieving representations that are disentangled, particularly for downstream tasks. However, there is some debate about how to encourage disentanglement with VAEs and evidence indicates that existing implementations of VAEs do not achieve disentanglement consistently. The evaluation of how well a VAE's latent space has been disentangled is often evaluated against our subjective expectations of which attributes should be disentangled for a given probl…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；NVIDIA（英伟达）
Textures and edges contribute different information to image recognition. Edges and boundaries encode shape information, while textures manifest the appearance of regions. Despite the success of Convolutional Neural Networks (CNNs) in computer vision and medical image analysis applications, predominantly only texture abstractions are learned, which often leads to imprecise boundary delineations. In medical imaging, expert manual segmentation often relies on organ boundaries; for example, to manually segment a liver, a medical practitioner usually identifies edges first and subsequently fills i…
TARDis: Time Attenuated Representation Disentanglement for Incomplete Multi-Modal Tumor Segmentation and Classification
School of Automation and Electrical Engineering, University of Science and Technology Beijing（北京科技大学自动化与电气工程学院）；Alibaba Group DAMO Academy（阿里巴巴集团DAMO学院）；Hupan Lab（湖畔实验室）；Departments of Radiology, Changhai Hospital（上海长海医院放射科）
Brain tumor segmentation is a challenging problem in medical image analysis. The endpoint is to generate the salient masks that accurately identify brain tumor regions in an fMRI screening. In this paper, we propose a novel attention gate (AG model) for brain tumor segmentation that utilizes both the edge detecting unit and the attention gated network to highlight and segment the salient regions from fMRI images. This feature enables us to eliminate the necessity of having to explicitly point towards the damaged area(external tissue localization) and classify(classification) as per classical c…
Uncertainty-Gated Deformable Network for Breast Tumor Segmentation in MR Images
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Information Engineering, Hefei University of Science（合肥科学技术大学计算机科学与信息工程学院）；Department of Radiology, Southwest Hospital, Army Medical University（西南医院放射科，中国人民解放军陆军军医大学）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）
Context-Gated Cross-Modal Perception with Visual Mamba for PET-CT Lung Tumor Segmentation
Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico di Roma, Italy（人工智能与计算机系统单位，罗马大学生物医学校园，意大利）；College of Computer Science and Software Engineering, Shenzhen University, China（计算机科学与软件工程学院，深圳大学，中国）；Dept. of Computer Engineering, Modeling, Electronic and System Engineering, University of Calabria, Italy（计算机工程、建模、电子与系统工程系，卡拉布里亚大学，意大利）；IRCCS San Raffaele Hospital, Italy（圣拉斐拉医院，意大利）；Faculty of Medicine, Vita-Salute San Raffaele University, Italy（医学学院，维塔-桑拉斐拉大学，意大利）；Dept. of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Sweden（诊断与干预系，辐射物理，生物医学工程，乌梅大学，瑞典）；Dept. of Naval, Electrical, Electronics and Telecommunications Engineering, University of Genoa, Italy（海军、电子、电子与电信工程系，热那亚大学，意大利）
Multi-modal MRIs are widely used in neuroimaging applications since different MR sequences provide complementary information about brain structures. Recent works have suggested that multi-modal deep learning analysis can benefit from explicitly disentangling anatomical (shape) and modality (appearance) information into separate image presentations. In this work, we challenge mainstream strategies by showing that they do not naturally lead to representation disentanglement both in theory and in practice. To address this issue, we propose a margin loss that regularizes the similarity in relation…
Tianjin University（天津大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；La Trobe University（乐卓博大学）；Tianjin University of Traditional Chinese Medicine（天津中医药大学）
The insufficiency of annotated medical imaging scans for cancer makes it challenging to train and validate data-hungry deep learning models in precision oncology. We propose a new richer generative adversarial network for free-form 3D tumor/lesion synthesis in computed tomography (CT) images. The network is composed of a new richer convolutional feature enhanced dilated-gated generator (RicherDG) and a hybrid loss function. The RicherDG has dilated-gated convolution layers to enable tumor-painting and to enlarge perceptive fields; and it has a novel richer convolutional feature association bra…
A Unified Attention U-Net Framework for Cross-Modality Tumor Segmentation in MRI and CT
Department of Computer Science, New Mexico State University（计算机科学系，新墨西哥州立大学）；Department of Plant and Environmental Sciences, New Mexico State University（植物与环境科学系，新墨西哥州立大学）
University of Electronic Science and Technology of China（电子科技大学）；Shanghai AI Lab（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；Sensetime Research（商汤科技研究院）；Sichuan University（四川大学）；Sichuan Cancer Hospital and Institute（四川省肿瘤医院（四川省肿瘤研究所））
Generalization to previously unseen images with potential domain shifts and different styles is essential for clinically applicable medical image segmentation, and the ability to disentangle domain-specific and domain-invariant features is key for achieving Domain Generalization (DG). However, existing DG methods can hardly achieve effective disentanglement to get high generalizability. To deal with this problem, we propose an efficient Contrastive Domain Disentanglement and Style Augmentation (CDDSA) framework for generalizable medical image segmentation. First, a disentangle network is propo…
Nanjing University（南京大学）；University of Wollongong（伍伦贡大学）；Nanjing Normal University（南京师范大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Due to low tissue contrast, irregular object appearance, and unpredictable location variation, segmenting the objects from different medical imaging modalities (e.g., CT, MR) is considered as an important yet challenging task. In this paper, we present a novel method for interactive medical image segmentation with the following merits. (1) Our design is fundamentally different from previous pure patch-based and image-based segmentation methods. We observe that during delineation, the physician repeatedly check the inside-outside intensity changing to determine the boundary, which indicates tha…
Massachusetts General Hospital（马萨诸塞总医院）；Harvard Medical School（哈佛医学院）
Multimodal MRI provides complementary and clinically relevant information to probe tissue condition and to characterize various diseases. However, it is often difficult to acquire sufficiently many modalities from the same subject due to limitations in study plans, while quantitative analysis is still demanded. In this work, we propose a unified conditional disentanglement framework to synthesize any arbitrary modality from an input modality. Our framework hinges on a cycle-constrained conditional adversarial training approach, where it can extract a modality-invariant anatomical feature with …
BaT-SAM: Introducing Inductive Bias into SAM via a Mixture of Convolutional Experts for Brain Tumor Segmentation
Mymensingh Medical College（迈门辛医学院）；West Virginia University（西弗吉尼亚大学）；Florida Polytechnic University（佛罗里达理工大学）；Manarat International University（马纳拉特国际大学）
School of Mathematics and Statistics, Changsha University of Science and Technology（数学与统计学学院，长沙理工大学）；School of Biomedical Engineering, Shanghai Jiao Tong University（生物医学工程学院，上海交通大学）；Shanghai Chest Hospital, Shanghai Jiao Tong University School of Medicine（上海胸科医院，上海交通大学医学院）
Multimodal Medical Endoscopic Image Analysis via Progressive Disentangle-aware Contrastive Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；First Affiliated Hospital, Sun Yat-sen University（中山大学第一附属医院）；College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）
Principled Feature Disentanglement for High-Fidelity Unified Brain MRI Synthesis
School of Computing, Korea Advanced Institute of Science（韩国科学技术院计算机学院）；Samsung Research, Seoul 06765, Republic of Korea（三星研究院）；Gordon Center for Medical Imaging, Massachusetts General Hospital（麻省总医院影像中心）；Harvard Medical School, Boston, MA 02114, USA（哈佛医学院）
BrainFusionNet: a deep learning and XAI model to understand local, global, and sequential features of MRI images for improved brain tumour detection
School of Mathematics, Physics and Computing, University of Southern Queensland（南方昆士兰大学数学、物理与计算学院）；School of Engineering, University of Southern Queensland（南方昆士兰大学工程学院）
Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT
University Health Network（大学健康网络）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Islamic Azad University（伊斯兰阿扎德大学）
Stanford University（斯坦福大学）；NVIDIA（英伟达）；National Institutes of Health（美国国立卫生研究院）
Multi-domain data are widely leveraged in vision applications taking advantage of complementary information from different modalities, e.g., brain tumor segmentation from multi-parametric magnetic resonance imaging (MRI). However, due to possible data corruption and different imaging protocols, the availability of images for each domain could vary amongst multiple data sources in practice, which makes it challenging to build a universal model with a varied set of input data. To tackle this problem, we propose a general approach to complete the random missing domain(s) data in real applications…
SINTEF（挪威科技工业研究院）；Bristol Royal Hospital for Children（布里斯托尔皇家儿童医院）；St. Olavs hospital（圣奥拉夫斯医院）
Meningiomas are the most common type of primary brain tumor, accounting for approximately 30% of all brain tumors. A substantial number of these tumors are never surgically removed but rather monitored over time. Automatic and precise meningioma segmentation is therefore beneficial to enable reliable growth estimation and patient-specific treatment planning. In this study, we propose the inclusion of attention mechanisms over a U-Net architecture: (i) Attention-gated U-Net (AGUNet) and (ii) Dual Attention U-Net (DAUNet), using a 3D MRI volume as input. Attention has the potential to leverage t…
Bridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
AI Technology for Life（人工智能技术与生命科学）；Department of Information and Computing Sciences（信息与计算科学系）；Department of Biology（生物学系）；Utrecht University（乌得勒支大学）；Department of Metabolic Diseases（代谢疾病部门）；Wilhelmina Children’s Hospital（维廉明娜儿童医院）；University Medical Center Utrecht（乌得勒支大学医学中心）；Regenerative Medicine Center Utrecht（乌得勒支再生医学中心）；Computational Pathology（计算病理学）；Department of Pathology（病理学系）；The Netherlands Cancer Institute（荷兰癌症研究所）；Visualization and Graphics（可视化与图形学）；The Netherlands Cancer Institute, Amsterdam（荷兰癌症研究所，阿姆斯特丹）
3D Lymphoma Segmentation on PET/CT Images via Multi-Scale Information Fusion with Cross-Attention
Zhengzhou University of Light Industry（郑州轻工业大学）；Peking University Institute of Hematology（北京大学血液病研究所）；Kennesaw State University（肯尼索州立大学）；Michigan Technological University（密歇根理工大学）
