MedMix：模态异质性下的专业化一致联邦稀疏混合专家模型

中文摘要 AI 辅助

联邦多模态医疗人工智能面临客户端和样本层面的模态异质性问题：客户端可能系统性地缺少特定模态类型，而同一客户端内的单个记录可能包含不同的部分模态子集。稀疏混合专家（MoE）架构是一种有前景的模态自适应计算解决方案，但在跨客户端模态异质性下，其在联邦学习中的应用较为脆弱，此时本地学习的路由策略会在不同客户端间产生分歧，导致专家形成不兼容的专业化方向。不同客户端可能将相同的观测模态配置分配给不同的专家，或在不同的缺失模态配置上训练索引相似的专家，使得标准聚合会错位或覆盖稀疏MoE旨在学习的专家专业化方向。为解决这一挑战，我们提出MedMix，一种用于联邦多模态稀疏MoE的语义对齐框架，其利用模态上下文协调跨客户端的路由和专家专业化。在客户端侧，MedMix使用模态上下文感知路由，通过每个 token 的模态身份、位置和不完整上下文指导专家选择。在客户端之间，它使用共识引导的路由对齐来构建服务器端共识锚点，以实现共享模态模式并对齐跨客户端的本地路由分布。作为这些路由机制的补充，客户端自适应专家聚合利用客户端特定的模态模式原型来匹配和聚合跨客户端功能相似的专家。在真实世界的多模态医疗数据集上的实验表明，MedMix在不同模态异质性和模态不完整设置下实现了最佳的平均F1值，在严重异质性下的提升尤为明显。

英文摘要

Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may contain different partial modality subsets. Sparse Mixture-of-Experts (MoE) architectures are a promising remedy for modality-adaptive computation, but their use in federated learning is fragile under cross-client modality heterogeneity, where locally learned routing policies can diverge across clients and drive experts toward incompatible specializations. Different clients may assign the same observed modality configuration to different experts, or train similarly indexed experts on different missing-modality configurations, causing standard aggregation to misalign or overwrite the expert specialization that sparse MoEs are intended to learn. To address this challenge, we propose MedMix, a semantic-alignment framework for federated multimodal sparse MoEs that coordinates cross-client routing and expert specialization using modality context. At the client side, MedMix uses modality-context-aware routing to guide expert selection using each token's modality identity, position, and incompleteness context. Across clients, it uses consensus-guided routing alignment to construct server-side consensus anchors for shared modality patterns and align local routing distributions across clients. Complementing these routing mechanisms, client-adaptive expert aggregation leverages client-specific modality-pattern prototypes to match and aggregate functionally similar experts across clients. Experiments on real-world multimodal medical datasets show that MedMix achieves the best average F1 across diverse modality heterogeneity and modality incompleteness settings, with especially clear gains under severe heterogeneity.

发表机构

• KAIST（韩国科学技术院）
• NTU Singapore（新加坡南洋理工大学）

机构由 AI 辅助整理，请以论文原文为准。