Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.
FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis
Shanghai University（上海大学）；North China Electric Power University（华北电力大学）；Université de Technologie de Compiègne（法国图卢兹国立理工学院）；Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；Macquarie University（麦考瑞大学）
Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；University of California, Irvine（加州大学欧文分校）；The Chinese University of Hong Kong（香港中文大学）
Federated Learning in Big Model Era: Domain-Specific Multimodal Large Models
ENN Group Co.,Ltd.（ENN集团有限公司）；China University of Petroleum（中国石油大学）；Unicom (Shanghai) Industrial Internet Co.,Ltd.（联通（上海）工业互联网有限公司）；Qingdao Windaka Technology Co.,Ltd.（青岛Windaka科技有限公司）
Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection
School of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程学院）；Department of Computer Science and Engineering, Yonsei University（延世大学计算机科学与工程系）；School of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程学院）；Saab Inc.（Saab公司）
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Institute of High-Performance Computing, A*STAR（新加坡高性能计算研究院（新加坡科研局下属机构））
Multimodal Federated Learning in Healthcare: a Review
West Virginia University（西弗吉尼亚大学）；Tribhuvan University（特里布万大学）；University at Albany（奥尔巴尼大学）；University of Pittsburgh（匹兹堡大学）；University of Aberdeen（阿伯丁大学）
MQFL-FHE: Multimodal Quantum Federated Learning Framework with Fully Homomorphic Encryption
SVKM's Dwarkadas J. Sanghvi College of Engineering（SVKM的达沃拉德·J·桑格维工程学院）；eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD)（eBRAIN实验室，工程系，纽约大学阿布扎比分校（NYUAD））；Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute（量子与拓扑系统中心（CQTS），NYUAD研究机构）；The Maersk Mc-Kinney Moller Institute, University of Southern Denmark（马士克·麦克金尼·莫勒研究所，南丹麦大学）；Corvinus Institute for Advanced Studies (CIAS), Budapest, Hungary（科维努斯高级研究学院（CIAS），布达佩斯，匈牙利）；Davidson School of Chemical Engineering, Purdue University（戴维森化学工程学院，普渡大学）
BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models
School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）；School of Computer Science and Technology, Anhui University（计算机科学与技术学院，安徽大学）；New Laboratory of Pattern Recognition (NLPR) State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Institute of Automation, Chinese Academy of Sciences (CASIA)（模式识别新实验室（NLPR）多模态人工智能系统国家重点实验室（MAIS）自动化研究所，中国科学院（CASIA））
Multimodal Federated Learning via Contrastive Representation Ensemble
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Waterloo（滑铁卢大学）；Carnegie Mellon University（卡内基梅隆大学）
Department of Biomedical Engineering, Linköping University, Sweden（_linköping大学生物医学工程系）；Department of Computer and Information Science, Linköping University, Sweden（_linköping大学计算机与信息科学系）；Center for Medical Image Science and Visualization (CMIV), Linköping University, Sweden（_linköping大学医学影像科学与可视化中心）
PID-Guided Partial Alignment for Multimodal Decentralized Federated Learning
Department of Electrical and Computer Engineering, Stony Brook University（石溪大学电气与计算机工程系）；Department of Applied Mathematics and Statistics and the Department of Computer Science, Stony Brook University（石溪大学应用数学与统计系和计算机科学系）；Department of Electrical and Computer Engineering, New York University（纽约大学电气与计算机工程系）；Department of Computer Science, New York Institute of Technology（纽约理工学院计算机科学系）
Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
University of Aberdeen（阿伯丁大学）；NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息技术研究所）；West Virginia University（西弗吉尼亚大学）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））
A Step Toward Federated Pretraining of Multimodal Large Language Models
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）
Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality
Kyung Hee University（庆熙大学）；The University of Danang—Vietnam-Korea University of Information and Communication Technology（岘港大学—越韩信息与通信技术大学）；Digital Science and Technology Institute（数字科学技术研究院）
Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Washington State University（华盛顿州立大学）；VinUniversity（文大学）；Griffin University（格里芬大学）；Hanoi University of Science and Technology（河内科学技术大学）
FedMultiEmo: Real-Time Emotion Recognition via Multimodal Federated Learning
Institute of Industrial Automation and Software Engineering, University of Stuttgart, Germany（工业自动化与软件工程研究所，斯图加特大学）；Electrical Engineering, University of Stuttgart, Germany（电气工程，斯图加特大学）；Graduate School of Excellence advanced Manufacturing Engineering, University of Stuttgart, Germany（先进制造工程优秀研究生院，斯图加特大学）
FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning
Institute for AI Industry Research, Tsinghua University（人工智能产业研究院，清华大学）；School of Software Engineering, Nankai University（软件工程学院，南开大学）；Department of Computing, Hong Kong Polytechnic University（计算机学院，香港理工大学）；AsiaInfo Technologies（亚信息科技）；China-Austria Belt and Road Joint Laboratory on Artificial Intelligence and Advanced Manufacturing, Hangzhou Dianzi University（人工智能与先进制造联合实验室，杭州电子科技大学）
FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation
Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University（浙江空间信息感知与传输重点实验室，杭州电子大学）；Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University（复杂系统建模与仿真实验室，计算机科学与技术学院，杭州电子大学）；Alibaba International Digital Commerce Group（阿里巴巴国际数字商业集团）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
M$^2$FedAQI: Multimodal Federated Learning for Air Quality Prediction on Heterogeneous Edge Devices
Department of Computer Science and Engineering, SRM University-AP, India（印度SRM大学-AP计算机科学与工程系）；Department of Mathematics and Computing, IIT (ISM) Dhanbad, India（印度德汉巴德IIT（ISM）数学与计算系）；Department of Mathematics and the Department of Computer Science, University of Central Florida, USA（美国中央佛罗里达大学数学系和计算机科学系）
Balancing Similarity and Complementarity for Federated Learning
Institute for Artificial Intelligence, Tsinghua University（清华大学人工智能研究院）；Tsinghua University（清华大学）；The University of Auckland（奥克兰大学）；RIKEN（理化学研究所）；Hong Kong Baptist University（香港浸会大学）；The University of Tokyo（东京大学）
ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis
School of Science and Engineering, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Department of Computer Science and Engineering, Lehigh University（利哈伊大学计算机科学与工程系）；Department of Computer Science, Emory University（埃默里大学计算机科学系）
