Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets.
Department of Electronic Engineering（电子工程系）；Fudan University（复旦大学）；School of Computing（计算学院）；Macquarie University（麦觉瑞大学）；Department of Communication Science and Engineering（通信科学与工程系）
FedSDAF: Leveraging Source Domain Awareness for Enhanced Federated Domain Generalization
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）；School of Computer Science and Technology, Tiangong University（天津工业大学计算机科学与技术学院）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
College of Intelligence and Computing, Tianjin University, Tianjin, China（智能与计算学院，天津大学，天津，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing
Parameter-Efficient and Personalized Federated Training of Generative Models at the Edge
Department of Computer Science, San Francisco State University（计算机科学系，圣弗朗西斯科州立大学）；Department of Communication Systems, University of Lakhimpur（通信系统系，拉贾斯坦邦大学）；Department of Embedded Systems, Manipur University（嵌入式系统系，曼尼普尔大学）；Department of Chemical Engineering, Lakshadweep University（化学工程系，拉克沙德韦普大学）
FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
School of Computing and Information Technology, Great Bay University（计算与信息学院，大亚湾大学）；Beijing Institute of Technology（北京理工大学）；University of Warwick（沃里克大学）；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；College of Computer Science and Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Shanghai AI Lab（上海人工智能实验室）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）；Ryerson University（瑞尔森大学）
FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization
School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；Manning College of Information & Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校曼宁信息与计算机科学学院）
Pennsylvania State University（宾夕法尼亚州立大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Purdue University（普渡大学）；Georgia State University（佐治亚州立大学）
Federated learning (FL) enables multiple clients to train models collaboratively without sharing local data, which has achieved promising results in different areas, including the Internet of Things (IoT). However, end IoT devices do not have abilities to automatically annotate their collected data, which leads to the label shortage issue at the client side. To collaboratively train an FL model, we can only use a small number of labeled data stored on the server. This is a new yet practical scenario in federated learning, i.e., labels-at-server semi-supervised federated learning (SemiFL). Alth…
Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learning
Hertie Institute for AI in Brain Health, University of Tübingen, Germany（人工智能与脑健康赫尔蒂研究所，图宾根大学，德国）；Tübingen AI Center, University of Tübingen, Germany（图宾根人工智能中心，图宾根大学，德国）
Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（高性能计算研究所（IHPC），科技研究局（A*STAR））；EVYD Technology（EVYD技术）
Peking University（北京大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Federated learning (FL) is an emerging promising paradigm of privacy-preserving machine learning (ML). An important type of FL is cross-silo FL, which enables a small scale of organizations to cooperatively train a shared model by keeping confidential data locally and aggregating weights on a central parameter server. However, the central server may be vulnerable to malicious attacks or software failures in practice. To address this issue, in this paper, we propose DeFL, a novel decentralized weight aggregation framework for cross-silo FL. DeFL eliminates the central server by aggregating weig…
Recently, Generative Adversarial Networks (GANs) have demonstrated their potential in federated learning, i.e., learning a centralized model from data privately hosted by multiple sites. A federatedGAN jointly trains a centralized generator and multiple private discriminators hosted at different sites. A major theoretical challenge for the federated GAN is the heterogeneity of the local data distributions. Traditional approaches cannot guarantee to learn the target distribution, which isa mixture of the highly different local distributions. This paper tackles this theoretical challenge, and fo…
Viterbi School of Engineering（维特比工程学院）；University of Southern California（南加州大学）
In Federated Learning, a common approach for aggregating local models across clients is periodic averaging of the full model parameters. It is, however, known that different layers of neural networks can have a different degree of model discrepancy across the clients. The conventional full aggregation scheme does not consider such a difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parameters that are similar across the clients does not make meaningful training progress while increasing the communication cost…
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
The Hong Kong Polytechnic University（香港理工大学）；OceanBase, Ant Group（蚂蚁集团OceanBase）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Mathematics and Statistics, Xidian University（西安电子科技大学数学与统计学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, China University of Geosciences（中国地质大学计算机学院）
Personalized Federated Sparse Adaptation of Time-Series Foundation Models
Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science（罗伯特·博世网络物理系统中心，印度科学学院）；School of Computing, Newcastle University（纽卡斯尔大学计算学院）
The federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server. Recently, adaptive optimization methods such as AdaGrad have been studied for server updates. However, the effect of using adaptive optimization methods for local updates at clients is not yet understood. We show in both theory and practice that while local adapt…
RefProtoFL: Communication-Efficient Federated Learning via External-Referenced Prototype Alignment
College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学）；State Key Lab. for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Yunnan Key Lab. of Service Computing, Yunnan University of Finance and Economics（云南服务计算重点实验室，云南财经大学）
