FedTaste：用于具有缺失模态的多模态联邦学习的拓扑感知结构转移
FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities
浏览论文内容
中文总结 AI 辅助
针对多模态联邦学习中模态缺失和数据分布问题，提出FedTaste框架，利用冻结基础模型提取联合多模态拓扑，结合模态自适应结构提示等方法，避免显式模态插补，在多数据集和非IID设置下性能优越且通信开销低。
中文摘要 AI 辅助
多模态联邦学习常面临任意模态缺失和非IID数据分布的挑战，导致严重的表示漂移并阻碍跨客户端的有效协作。现有方法存在诸多问题。本文提出FedTaste，一个用于具有缺失模态的多模态联邦学习的拓扑感知结构转移的参数高效框架。它利用冻结基础模型提取联合多模态拓扑，通过模态自适应结构提示和谱一致性正则化实现客户端适应，避免显式模态插补并保留共享语义结构。实验表明其性能优越且通信开销低。
英文摘要
Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing methods typically rely on generative imputation, external auxiliary data, or isolated unimodal training to bridge modality gaps, often incurring substantial communication and computational costs as well as potential privacy risks. To address these limitations, we propose FedTaste, a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning with missing modalities. Instead of aligning fragile first-order features, FedTaste focuses on more stable group-level semantic relations. Specifically, FedTaste leverages frozen foundation models to extract a joint multimodal topology from full-modality clients, which is then consolidated by the server into a global structural blueprint. To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts together with spectral consistency regularization, enabling lightweight branch-specific adaptation that aligns local partial representations with the shared blueprint. In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients. Extensive experiments demonstrate that FedTaste consistently achieves superior performance across multiple datasets and challenging Non-IID settings, while substantially reducing communication overhead compared with existing methods.
发表机构
- The University of Tokyo（东京大学）
- Great Bay University（大湾区大学）
机构由 AI 辅助整理，请以论文原文为准。
