你在看什么？多模态医学深度学习中的模态贡献
What are You Looking at? Modality Contribution in Multimodal Medical Deep Learning
AI总结：
针对多模态医学深度学习模型的模态信息处理机制不明确的问题，提出一种与模型、性能无关的基于遮挡的模态贡献定量方法，经三个医学任务实验发现单模态崩塌、数据集不平衡等问题，为模型开发与数据集构建提供支持，助力可解释性研究与临床应用。
摘要：
研究目的 如今，大型深度神经网络可轻松分析高维多模态数据。目前已开发出多种融合不同模态的方法。鉴于医学领域高维多模态患者数据的普遍性，多模态模型的发展是一项重大进步。然而，这些模型如何详细处理来自各个来源的信息仍有待深入探究。研究方法 为此，我们实现了一种基于遮挡的模态贡献方法，该方法与模型及性能均无关，可定量衡量数据集中每个模态对模型完成任务的重要性。我们将该方法应用于三个不同的多模态医学问题以开展实验。研究结果 研究发现，部分网络存在模态偏好，易出现单模态崩塌现象，而部分数据集从根本上存在不平衡问题。此外，我们还提供了每个模态的细粒度定量及可视化属性重要性。研究结论 我们的指标提供了宝贵见解，可助力多模态模型开发与数据集构建的推进。通过引入该方法，我们为多模态研究中日益发展的深度学习可解释性领域做出了贡献。该方法有助于推动多模态AI融入临床实践。我们的代码已公开于https://github.com/ChristianGappGit/MC_MMD。
英文摘要：
Purpose High dimensional, multimodal data can nowadays be analyzed by huge deep neural networks with little effort. Several fusion methods for bringing together different modalities have been developed. Given the prevalence of high-dimensional, multimodal patient data in medicine, the development of multimodal models marks a significant advancement. However, how these models process information from individual sources in detail is still underexplored. Methods To this end, we implemented an occlusion-based modality contribution method that is both model- and performance-agnostic. This method quantitatively measures the importance of each modality in the dataset for the model to fulfill its task. We applied our method to three different multimodal medical problems for experimental purposes. Results Herein we found that some networks have modality preferences that tend to unimodal collapses, while some datasets are imbalanced from the ground up. Moreover, we provide fine-grained quantitative and visual attribute importance for each modality. Conclusion Our metric offers valuable insights that can support the advancement of multimodal model development and dataset creation. By introducing this method, we contribute to the growing field of interpretability in deep learning for multimodal research. This approach helps to facilitate the integration of multimodal AI into clinical practice. Our code is publicly available at https://github.com/ChristianGappGit/MC_MMD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
