HyperMM:面向可变尺寸输入的鲁棒多模态学习
HyperMM : Robust Multimodal Learning with Varying-sized Inputs
AI总结:
提出HyperMM端到端框架,利用条件超网络和置换不变神经网络处理缺失模态的可变尺寸输入,无需预填补即可实现鲁棒的多模态学习,在病理诊断任务中表现出色。
摘要:
通过多模态学习(MML)结合携带互补信息的多种模态,已显示出对多种病理诊断的显著益处。然而,多模态模型对缺失模态的鲁棒性经常被忽视。大多数工作假设输入数据中的模态是完整的,而在临床实践中,模态不完整的情况很常见。解决这一问题的现有解决方案依赖于在使用监督学习模型之前进行模态填补策略。然而,这些策略复杂、计算成本高,并且可能对后续预测模型产生强烈影响。因此,在医疗保健等敏感应用中应谨慎使用它们。我们提出了HyperMM,这是一个为可变尺寸输入学习而设计的端到端框架。具体而言,我们专注于在训练前不使用填补的缺失成像模态监督MML任务。我们引入了一种使用条件超网络训练通用特征提取器的新策略,并提出了一种置换不变神经网络,该网络能够处理不同维度的输入以处理提取的特征,处于一个两阶段的任务无关框架中。我们在两项任务中实验性地证明了我们方法的优势:阿尔茨海默病检测和乳腺癌分类。我们证明我们的策略对高缺失率数据具有鲁棒性,并且其灵活性使其能够处理可变尺寸数据集,而不仅限于缺失模态的场景。
英文摘要:
Combining multiple modalities carrying complementary information through multimodal learning (MML) has shown considerable benefits for diagnosing multiple pathologies. However, the robustness of multimodal models to missing modalities is often overlooked. Most works assume modality completeness in the input data, while in clinical practice, it is common to have incomplete modalities. Existing solutions that address this issue rely on modality imputation strategies before using supervised learning models. These strategies, however, are complex, computationally costly and can strongly impact subsequent prediction models. Hence, they should be used with parsimony in sensitive applications such as healthcare. We propose HyperMM, an end-to-end framework designed for learning with varying-sized inputs. Specifically, we focus on the task of supervised MML with missing imaging modalities without using imputation before training. We introduce a novel strategy for training a universal feature extractor using a conditional hypernetwork, and propose a permutation-invariant neural network that can handle inputs of varying dimensions to process the extracted features, in a two-phase task-agnostic framework. We experimentally demonstrate the advantages of our method in two tasks: Alzheimer's disease detection and breast cancer classification. We demonstrate that our strategy is robust to high rates of missing data and that its flexibility allows it to handle varying-sized datasets beyond the scenario of missing modalities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。