Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models." In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.
Exploring Vacant Classes in Label-Skewed Federated Learning
University of Science and Technology of China(中国科学技术大学);Anhui University(安徽大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing University(南京大学)
Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation
Institute of Information & Communications Technology Planning & Evaluation (IITP)-ITRC (Information Technology Research Center)(信息与通信技术规划与评估机构(IITP)-ITRC(信息技术研究中心))
CoM$^3$eT: A foundation model for medical image analysis through federated, multidimensional context integration
Fraunhofer Institute for Digital Medicine MEVIS(弗劳恩霍夫数字医学MEVIS研究所);RWTH Aachen University(亚琛工业大学);Medizinische Hochschule Hannover(汉诺威医学院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Charité – Universitätsmedizin Berlin(柏林夏里特医学院);Freie Universität Berlin(柏林自由大学);Humboldt-Universität zu Berlin(柏林洪堡大学)
Exploring Machine Learning Models for Federated Learning: A Review of Approaches, Performance, and Limitations
Data Science and Analytics Institute, University of Oklahoma(俄克拉荷马大学数据科学与分析学院);School of Industrial and Systems Engineering, University of Oklahoma(俄克拉荷马大学工业与系统工程学院);Isfahan University of Technology(伊斯法罕理工大学)
A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendation
Australian Artificial Intelligence Institute, University of Technology Sydney(澳大利亚人工智能研究所,悉尼技术大学);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系)
GraphDLG: Exploring Deep Leakage from Gradients in Federated Graph Learning
C-FAIR \& School of Software Shandong University Jinan China;SKLCCSE Lab Beihang University Beijing China;Automation \& Intelligent Sensing Shanghai Jiao Tong University Shanghai China;Shandong University;Beihang University;Shanghai Jiao Tong University
Synthetic Data Aided Federated Learning Using Foundation Models
The University of Manchester(曼彻斯特大学);Institute for Infocomm Research, A*STAR(新加坡资讯通信研究院(新加坡科技研究局下属机构));KU Leuven(鲁汶大学);COSIC, KU Leuven(鲁汶大学计算机与安全工业密码学中心)
FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Future Network of Intelligence Institute(未来智联网络研究院);Guangdong Provincial Key Laboratory of Future Networks of Intelligence(广东省未来智联网络重点实验室)
ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning
Tsinghua University(清华大学);Microsoft Research(微软研究院);Idiap Research Institute(Idiap研究所);UIUC(伊利诺伊大学厄巴纳-香槟分校);Chinese Academy of Sciences(中国科学院);Hong Kong University of Science and Technology(香港科技大学)
DP2FL: Dual Prompt Personalized Federated Learning in Foundation Models
College of Software, Jilin University(吉林大学软件学院);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);College of Computer Science and Engineering, Changchun University of Technology(长春理工大学计算机科学与工程学院)
Exploring New Frontiers in Vertical Federated Learning: the Role of Saddle Point Reformulation
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)(人工智能基础研究实验室(BRAIn实验室));Federated Learning Problems Laboratory(联邦学习问题实验室);Innopolis University(Innopolis大学);Sber AI Lab(俄联邦储蓄银行人工智能实验室);Technical University of Munich(慕尼黑工业大学);HSE University(高等经济大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Discrete Prototypical Memories for Federated Time Series Foundation Models
Australian Artificial Intelligence Institute, University of Technology Sydney(澳大利亚人工智能研究所,悉尼科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shenzhen University(深圳大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Sichuan University(四川大学)
Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputers
Mathematics and Computer Science Division, Argonne National Laboratory(阿贡国家实验室数学与计算机科学 division);Data Science and Learning Division, Argonne National Laboratory(阿贡国家实验室数据科学与学习 division);Department of Computer Science, University of Chicago(芝加哥大学计算机科学系)
Synergizing Foundation Models and Federated Learning: A Survey
Uppsala University(乌普萨拉大学);University College London(伦敦大学学院);University of Liverpool(利物浦大学);Eindhoven University of Technology(埃因霍温理工大学);The University of Hong Kong(香港大学);Research Institutes of Sweden(瑞典研究机构)