Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network architecture that maps a variable number of input modalities into a unified representation that can be used for downstream tasks such as segmentation. We demonstrate that modality dropout makes a standard segmentation network reasonably robust to missing modalities, but that the same network works even better if trained on the unified representation.
Beijing-Dublin International College（北京-都柏林国际学院）；University College Dublin（都柏林大学）
This technical report presents a comparative analysis of existing deep learning (DL) based approaches for brain tumor segmentation with missing MRI modalities. Approaches evaluated include the Adversarial Co-training Network (ACN) and a combination of mmGAN and DeepMedic. A more stable and easy-to-use version of mmGAN is also open-sourced at a GitHub repository. Using the BraTS2018 dataset, this work demonstrates that the state-of-the-art ACN performs better especially when T1c is missing. While a simple combination of mmGAN and DeepMedic also shows strong potentials when only one MRI modality…
Robust Brain Tumor Segmentation with Incomplete MRI Modalities Using Hölder Divergence and Mutual Information-Enhanced Knowledge Transfer
MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University（深圳MSU-BIT大学应用数学联合研究中心）；Institute of Control Theory and Control Engineering, School of Automation, Beijing Institute of Technology（北京理工大学控制理论与控制工程研究所）；School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
University Medical Center Utrecht（乌得勒支大学医学中心）；Eindhoven University of Technology（埃因霍温理工大学）
Automatic segmentation of medical images is an important task for many clinical applications. In practice, a wide range of anatomical structures are visualised using different imaging modalities. In this paper, we investigate whether a single convolutional neural network (CNN) can be trained to perform different segmentation tasks. A single CNN is trained to segment six tissues in MR brain images, the pectoral muscle in MR breast images, and the coronary arteries in cardiac CTA. The CNN therefore learns to identify the imaging modality, the visualised anatomical structures, and the tissue clas…
INSA Rouen（鲁昂国立应用科学学院）；Université de Rouen Normandie（鲁昂大学）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it's common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent th…
Deep learning models, such as the fully convolutional network (FCN), have been widely used in 3D biomedical segmentation and achieved state-of-the-art performance. Multiple modalities are often used for disease diagnosis and quantification. Two approaches are widely used in the literature to fuse multiple modalities in the segmentation networks: early-fusion (which stacks multiple modalities as different input channels) and late-fusion (which fuses the segmentation results from different modalities at the very end). These fusion methods easily suffer from the cross-modal interference caused by…
RWTH Aachen University（亚琛工业大学）；Iran University of Science and Technology（伊朗科技大学）；University of Montreal（蒙特利尔大学）；Polytechnique Montreal（蒙特利尔综合理工学院）；Mila, Quebec AI Institute（魁北克人工智能研究所Mila）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）
Dealing with missing modalities in Magnetic Resonance Imaging (MRI) and overcoming their negative repercussions is considered a hurdle in biomedical imaging. The combination of a specified set of modalities, which is selected depending on the scenario and anatomical part being scanned, will provide medical practitioners with full information about the region of interest in the human body, hence the missing MRI sequences should be reimbursed. The compensation of the adverse impact of losing useful information owing to the lack of one or more modalities is a well-known challenge in the field of …
impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction
Center for Applied AI, Skolkovo Institute of Science and Technology, Moscow, Russian Federation（应用人工智能中心，斯克尔科夫科学与技术研究所，莫斯科，俄罗斯联邦）；BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory, University of Sharjah, Sharjah, United Arab Emirates（BIMAI实验室，生物医学导向人工智能实验室，肖贾联合大学，肖贾，阿拉伯联合酋长国）；Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russian Federation（伊万诺夫俄罗斯科学院系统编程研究所，莫斯科，俄罗斯联邦）
Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training
Department of Engineering Science, University of Oxford, Oxford, UK（工程科学系，牛津大学，牛津，英国）；Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK（牛津大学临床神经科学系，牛津，英国）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific mo…
TU Kaiserslautern（凯泽斯劳滕工业大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
The integration of information acquired with different modalities, spatial resolution and spectral bands has shown to improve predictive accuracies. Data fusion is therefore one of the key challenges in remote sensing. Most prior work focusing on multi-modal fusion, assumes that modalities are always available during inference. This assumption limits the applications of multi-modal models since in practice the data collection process is likely to generate data with missing, incomplete or corrupted modalities. In this paper, we show that Generative Adversarial Networks can be effectively used t…
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳香槟分校联合学院）；Zhejiang University（浙江大学）
Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3…
Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology Taxila（塔克西拉工程大学）；Dalhousie University（达尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心）；Johannes Kepler University Linz（约翰开普勒林茨大学）；Linz Institute of Technology（林茨理工学院）
UiT The Arctic University of Norway（挪威北极大学）；Norwegian Computing Center（挪威计算中心）
Automatic urban land cover classification is a fundamental problem in remote sensing, e.g. for environmental monitoring. The problem is highly challenging, as classes generally have high inter-class and low intra-class variance. Techniques to improve urban land cover classification performance in remote sensing include fusion of data from different sensors with different data modalities. However, such techniques require all modalities to be available to the classifier in the decision-making process, i.e. at test time, as well as in training. If a data modality is missing at test time, current …
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）
Brain Tumor Segmentation from magnetic resonance imaging (MRI) is a critical technique for early diagnosis. However, rather than having complete four modalities as in BraTS dataset, it is common to have missing modalities in clinical scenarios. We design a brain tumor segmentation algorithm that is robust to the absence of any modality. Our network includes a channel-independent encoding path and a feature-fusion decoding path. We use self-supervised training through channel dropout and also propose a novel domain adaptation method on feature maps to recover the information from the missing ch…
Synergistic Prompting for Robust Visual Recognition with Missing Modalities
Beijing Institute of Technology（北京理工大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Systems Laboratory of Anhui Province, Hefei University of Technology（安徽省系统实验室，合肥工业大学）；National Innovative Institute of Defense Technology（国家创新防御技术研究院）；Intelligent Science & Technology Academy of CASIC（中国航天科技集团智能科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
IMT School for Advanced Studies of Lucca（卢卡高等研究院）；Institute of Computational Perception, Johannes Kepler University（约翰开普勒大学计算感知研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology, Taxila（塔克西拉工程技术大学）；Dalhousie University（戴尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（微电子学研究中心）；Linz Institute of Technology（林茨理工学院）
National Yang Ming Chiao Tung University（国立阳明交通大学）；Google（谷歌）
In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire mod…
Robust Multi-Modal Face Anti-Spoofing with Domain Adaptation: Tackling Missing Modalities, Noisy Pseudo-Labels, and Model Degradation
College of Computer and Cyber Security, Fujian Normal University（计算机与网络安全部，福建师范大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China（东部理工学院计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University, China（上海大学机电工程与自动化学院）
Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
School of Mathematics, Shandong University（山东大学数学学院）；GSFS, The University of Tokyo（东京大学GSFS）；MedVisAI Lab（医学视觉人工智能实验室）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光前医学院）；Centre of AI in Medicine, Singapore（新加坡医学人工智能中心）
University of Delaware（特拉华大学）；Snap Inc.（Snap公司）；Rutgers University（罗格斯大学）
A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., 90% training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal lear…
CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Aberdeen（阿伯丁大学）
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China)（青岛软件研究所，计算机科学与技术学院，中国石油大学（华东））；Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software（山东省智能油气工业软件重点实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Harvard University（哈佛大学）；Technical University of Munich（慕尼黑工业大学）；MIT（麻省理工学院）；University of Cambridge（剑桥大学）；Donostia International Physics Center（多诺斯蒂亚国际物理中心）；University of the Basque Country（巴斯克大学）；Ikerbasque, Basque Foundation for Science（巴斯克科学基金会伊克巴斯克）；HHMI（霍华德·休斯医学研究所）；Boston College（波士顿学院）
Instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming. Existing works segment a new modality by either deploying a pre-trained model optimized on diverse training data or conducting domain translation and image segmentation as two independent steps. In this work, we propose a novel Cyclic Segmentation Generative Adversarial Network (CySGAN) that conducts image translation and instance segmentation jointly using a unified framework. Besides the CycleGAN losses for image translation and super…
Imagia Inc.（Imagia 公司）；Université de Sherbrooke（舍布鲁克大学）；Université de Montréal（蒙特利尔大学）
We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, …
Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to predict the missing data across modalities, the inherent difference between heterogeneous modalities, namely the modality gap, presents a challenge. To address this, we propose to use invariant features for a missing modality imagination network (IF-MMIN) which includes two novel mechanisms: 1) an invariant feature learning strategy that is based on the central moment discrepancy …
Missing data is a common problem in machine learning and in retrospective imaging research it is often encountered in the form of missing imaging modalities. We propose to take into account missing modalities in the design and training of neural networks, to ensure that they are capable of providing the best possible prediction even when multiple images are not available. The proposed network combines three modifications to the standard 3D UNet architecture: a training scheme with dropout of modalities, a multi-pathway architecture with fusion layer in the final stage, and the separate pre-tra…
School of Informatics, Xiamen University（厦门大学信息学院）；Tencent Jarvis Lab, Tencent Healthcare (Shenzhen) Co., Ltd.（腾讯医疗健康（深圳）有限公司腾讯杰维斯实验室）；School of Medicine, Xiamen University（厦门大学医学院）
Multimodal magnetic resonance imaging (MRI) provides complementary information for sub-region analysis of brain tumors. Plenty of methods have been proposed for automatic brain tumor segmentation using four common MRI modalities and achieved remarkable performance. In practice, however, it is common to have one or more modalities missing due to image corruption, artifacts, acquisition protocols, allergy to contrast agents, or simply cost. In this work, we propose a novel two-stage framework for brain tumor segmentation with missing modalities. In the first stage, a multimodal masked autoencode…
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
University of Electronic Science and Technology of China（电子科技大学）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
PRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China（生物医学工程系，香港理工大学，香港特别行政区，中国）；Department of Technology Management for Innovation, The University of Tokyo, Japan（创新技术管理系，东京大学，日本）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China（数据科学与人工智能系，香港理工大学，香港特别行政区，中国）
Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning
Johannes Kepler University（约翰内斯·开普勒大学）；Aalto University（阿alto大学）；University of Michigan-Flint（密歇根大学弗林特分校）；Macquarie University（麦考瑞大学）；Institute of Computational Perception, Johannes Kepler University Linz（计算感知研究所，约翰内斯·开普勒大学林茨）；Linz Institute of Technology（林茨技术研究所）
