检查多模态联邦学习中用于医学视觉和基于语言的疾病检测的模态不一致性
Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection
AI总结：
本文首次分析多模态联邦学习中模态不一致性的影响，提出三种缓解途径（自注意力、模态插补网络、正则化），并在两个真实数据集上验证其优于单模态联邦学习。
摘要：
多模态联邦学习（MMFL）利用每个客户端中的多种模态来构建比单模态联邦学习（FL）更强大的模型。然而，不同客户端中缺失模态的影响，也称为模态不一致性，在很大程度上被忽视了。本文首次分析了模态不一致性的影响，并揭示了其与参与客户端之间数据异质性的联系。我们特别检查了具有单模态和多模态客户端的不一致MMFL是否比单模态FL更有益。此外，我们探讨了解决这一问题的三种潜在途径。首先，我们研究了各种自注意力机制在MMFL中对不一致性无关信息融合的有效性。其次，我们引入了一个在多模态客户端中预训练的模态插补网络（MIN），用于单模态客户端的模态转换，并研究其在缓解缺失模态问题方面的潜力。第三，我们评估了客户端级和服务器级正则化技术在缓解模态不一致性影响方面的能力。实验在两种公开的真实世界数据集MIMIC-CXR和Open-I上，使用胸部X射线和放射学报告，在多种MMFL设置下进行。
英文摘要：
Multimodal Federated Learning (MMFL) utilizes multiple modalities in each client to build a more powerful Federated Learning (FL) model than its unimodal counterpart. However, the impact of missing modality in different clients, also called modality incongruity, has been greatly overlooked. This paper, for the first time, analyses the impact of modality incongruity and reveals its connection with data heterogeneity across participating clients. We particularly inspect whether incongruent MMFL with unimodal and multimodal clients is more beneficial than unimodal FL. Furthermore, we examine three potential routes of addressing this issue. Firstly, we study the effectiveness of various self-attention mechanisms towards incongruity-agnostic information fusion in MMFL. Secondly, we introduce a modality imputation network (MIN) pre-trained in a multimodal client for modality translation in unimodal clients and investigate its potential towards mitigating the missing modality problem. Thirdly, we assess the capability of client-level and server-level regularization techniques towards mitigating modality incongruity effects. Experiments are conducted under several MMFL settings on two publicly available real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology reports.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
