MIND：面向多模态临床预测任务的模态信息知识蒸馏框架
MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks
AI总结：
本文提出MIND框架，通过知识蒸馏将多个单模态教师网络的知识迁移到小型多模态学生模型，并采用多头联合融合处理缺失模态，在临床预测及非医学多模态任务上提升小模型性能。
摘要：
多模态融合利用跨模态信息来学习更好的特征表示，以提高基于融合的任务的性能。然而，多模态数据集，尤其是在医学环境中，通常比其单模态对应数据集更小，这可能阻碍多模态模型的性能。此外，模态数量的增加通常伴随着多模态网络整体规模的增大，这在医疗用例中可能并不理想。使用较小的单模态编码器可能导致次优性能，尤其是在处理高维临床数据时。在本文中，我们提出了模态信息知识蒸馏（MIND）框架，这是一种基于知识蒸馏的多模态模型压缩方法，将来自不同规模的预训练深度神经网络集成体的知识迁移到较小的多模态学生模型中。教师模型由单模态网络组成，使学生能够从多样化的表示中学习。MIND采用多头联合融合模型，而非单头模型，从而在单模态样本的情况下能够使用单模态编码器，而无需对缺失模态进行插补或掩码。因此，MIND生成了一个优化的多模态模型，同时增强了多模态和单模态表示。它还可以用于在训练过程中平衡多模态学习。我们使用时序数据和胸部X光图像，在二分类和多标签临床预测任务上评估MIND。此外，我们在三个非医学多模态多类数据集上评估了MIND框架的泛化能力。实验结果表明，与最先进的基线方法相比，MIND在所有五个任务以及各种融合方法和多模态架构上均提升了较小多模态网络的性能。
英文摘要：
Multimodal fusion leverages information across modalities to learn better feature representations with the goal of improving performance in fusion-based tasks. However, multimodal datasets, especially in medical settings, are typically smaller than their unimodal counterparts, which can impede the performance of multimodal models. Additionally, the increase in the number of modalities is often associated with an overall increase in the size of the multimodal network, which may be undesirable in medical use cases. Utilizing smaller unimodal encoders may lead to sub-optimal performance, particularly when dealing with high-dimensional clinical data. In this paper, we propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach based on knowledge distillation that transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student. The teacher models consist of unimodal networks, allowing the student to learn from diverse representations. MIND employs multi-head joint fusion models, as opposed to single-head models, enabling the use of unimodal encoders in the case of unimodal samples without requiring imputation or masking of absent modalities. As a result, MIND generates an optimized multimodal model, enhancing both multimodal and unimodal representations. It can also be leveraged to balance multimodal learning during training. We evaluate MIND on binary and multilabel clinical prediction tasks using time series data and chest X-ray images. Additionally, we assess the generalizability of the MIND framework on three non-medical multimodal multiclass datasets. Experimental results demonstrate that MIND enhances the performance of the smaller multimodal network across all five tasks, as well as various fusion methods and multimodal architectures, compared to state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
