Multimodal Federated Learning (MFL) with mixed modalities enables unimodal and multimodal clients to collaboratively train models while ensuring clients' privacy. As a representative sample of local data, prototypes offer an approach with low resource consumption and no reliance on prior knowledge for MFL with mixed modalities. However, existing prototype-based MFL methods assume unified labels across clients and identical tasks per client, which is impractical in MFL with mixed modalities. In this work, we propose an Adaptive prototype-based Multimodal Federated Learning (AproMFL) framework for mixed modalities to address the aforementioned issues. Our AproMFL transfers knowledge through adaptively-constructed prototypes without unified labels. Clients adaptively select prototype construction methods in line with labels; server converts client prototypes into unified multimodal prototypes and cluster them to form global prototypes. To address model aggregation issues in task heterogeneity, we develop a client relationship graph-based scheme to dynamically adjust aggregation weights. Furthermore, we propose a global prototype knowledge transfer loss and a global model knowledge transfer loss to enable the transfer of global knowledge to local knowledge. Experimental results show that AproMFL outperforms four baselines on three highly heterogeneous datasets ($α=0.1$) and two heterogeneous tasks, with the optimal results in accuracy and recall being 0.42%~6.09% and 1.6%~3.89% higher than those of FedIoT (FedAvg-based MFL), respectively.
Mitigating Domain Shift in Federated Learning via Intra- and Inter-Domain Prototypes
Department of Computer Science and Engineering, School of Computing, Kyung Hee University（计算机科学与工程系，计算学院，庆熙大学）；Department of Artificial Intelligence, School of Computing, Kyung Hee University（人工智能系，计算学院，庆熙大学）；Digital Science and Technology Institute, The University of Danang—Vietnam-Korea University of Information and Communication Technology（数字科学与技术研究所，丹那大学—越南-韩国信息与通信技术大学）
Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing, China（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学，北京，中国）；Zhongguancun Laboratory, Beijing, China（中关村实验室，北京，中国）；Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo, USA（人工智能商业创新中心，管理科学与系统系，管理学院，布法罗大学，美国）
Robust Federated Learning on Edge Devices with Domain Heterogeneity
Department of Computer Science and Engineering, Kyung Hee University, 171-04, Republic of Korea（韩国庆熙大学计算机科学与工程系）；Machine learning department, Mohamed Bin Zayed University of Artificial Intelligence, United Arab Emirates（阿联酋穆罕默德·本·扎耶德人工智能大学机器学习系）
Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
University of Aberdeen（阿伯丁大学）；NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息技术研究所）；West Virginia University（西弗吉尼亚大学）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））
RefProtoFL: Communication-Efficient Federated Learning via External-Referenced Prototype Alignment
College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学）；State Key Lab. for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Yunnan Key Lab. of Service Computing, Yunnan University of Finance and Economics（云南服务计算重点实验室，云南财经大学）
Personalized federated prototype learning in mixed heterogeneous data scenarios
Guangxi Normal University（广西师范大学）；Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, GuangXi Normal University（教育区块链与智能技术重点实验室）；Hefei University of Technology（合肥工业大学）
Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo（人工智能商业创新中心，管理科学与系统系，布法罗大学）
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越韩信息与通信技术大学）；Digital Science and Technology Institute（数字科学技术研究院）
Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learning
Hertie Institute for AI in Brain Health, University of Tübingen, Germany（人工智能与脑健康赫尔蒂研究所，图宾根大学，德国）；Tübingen AI Center, University of Tübingen, Germany（图宾根人工智能中心，图宾根大学，德国）
Proto-EVFL: Enhanced Vertical Federated Learning via Dual Prototype with Extremely Unaligned Data
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；Center for Applied Statistics, School of Statistics, Renmin University of China（中国人民大学统计学院应用统计中心）；Beijing Academy of Blockchain and Edge Computing（北京区块链与边缘计算研究院）；Department of Statistics, Fudan University（复旦大学统计系）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
FedPall: Prototype-based Adversarial and Collaborative Learning for Federated Learning with Feature Drift
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳MSU-BIT大学人工智能研究院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）
Cross-Modal Vertical Federated Learning for MRI Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Jarvis Lab（腾讯Jarvis实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；University of California, Irvine（加州大学欧文分校）；The Chinese University of Hong Kong（香港中文大学）
FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization
Hanoi University of Science and Technology（河内科学技术大学）；Nagoya University（名古屋大学）；Washington State University（华盛顿州立大学）；Swiss Federal Institute of Technology Lausanne（洛桑联邦理工学院）
Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection
The Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
