Federated learning (FL) is a promising approach to distributed compute, as well as distributed data, and provides a level of privacy and compliance to legal frameworks. This makes FL attractive for both consumer and healthcare applications. While the area is actively being explored, few studies have examined FL in the context of larger language models and there is a lack of comprehensive reviews of robustness across tasks, architectures, numbers of clients, and other relevant factors. In this paper, we explore the fine-tuning of Transformer-based language models in a federated learning setting. We evaluate three popular BERT-variants of different sizes (BERT, ALBERT, and DistilBERT) on a number of text classification tasks such as sentiment analysis and author identification. We perform an extensive sweep over the number of clients, ranging up to 32, to evaluate the impact of distributed compute on task performance in the federated averaging setting. While our findings suggest that the large sizes of the evaluated models are not generally prohibitive to federated training, we found that the different models handle federated averaging to a varying degree. Most notably, DistilBERT converges significantly slower with larger numbers of clients, and under some circumstances, even collapses to chance level performance. Investigating this issue presents an interesting perspective for future research.
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
We describe the design of our federated task processing system. Originally, the system was created to support two specific federated tasks: evaluation and tuning of on-device ML systems, primarily for the purpose of personalizing these systems. In recent years, support for an additional federated task has been added: federated learning (FL) of deep neural networks. To our knowledge, only one other system has been described in literature that supports FL at scale. We include comparisons to that system to help discuss design decisions and attached trade-offs. Finally, we describe two specific la…
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage various techniques for mitigating these bottlenecks to train larger language models in cross-device federated learning. With systematic applications of partial model training, quantization, efficient transfer learning, and communication-efficient optimizers, we are able to train a $21$M parameter Transformer and $20.2$M parameter Conformer that achieve the same or better perplexity as that of a similarly sized LSTM with $…
Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data
Washington State University（华盛顿州立大学）；Princeton University（普林斯顿大学）；IBM Research（IBM研究院）；University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣迭戈分校）
University of California, Davis（加州大学戴维斯分校）；The Ohio State University（俄亥俄州立大学）；Michigan State University（密歇根州立大学）；DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）
Federated learning (FL) hyper-parameters significantly affect the training overheads in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters puts a high burden on FL practitioners since various applications prefer different training preferences. In this paper, we propose FedTune, an automatic FL hyper-parameter tuning algorithm tailored to applications' diverse system requirements of FL training. FedTune is lightweight and flexible, achieving 8.48%-26.75% improvement for different datasets…
Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning
CISE, University of Florida, FL, USA（佛罗里达大学计算机与信息科学系）；North Carolina State University, NC, USA（北卡罗来纳州立大学）；Washington State University, WA, USA（华盛顿州立大学）；Seoul National University of Science and Technology, South Korea（韩国科学技术大学）
Information Sciences Institute（信息科学研究所）；University of Southern California（南加州大学）；Imaging Genetics Center（影像遗传学中心）；USC Stevens Neuroimaging and Informatics Institute（南加州大学史蒂文斯神经影像与信息学研究所）
The amount of biomedical data continues to grow rapidly. However, the ability to analyze these data is limited due to privacy and regulatory concerns. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning is a promising approach to learn a joint model over data silos. This architecture does not share any subject data across sites, only aggregated parameters, often in encrypted environments, thus satisfying privacy and regulatory requirements. Here, we describe our Federated Learning architecture and tr…
Shandong University（山东大学）；Bank of Jiangsu（江苏银行）；Fudan University（复旦大学）；Ohio State University（俄亥俄州立大学）；University of California, Davis（加州大学戴维斯分校）
Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose FedT…
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Institute of High-Performance Computing, A*STAR（新加坡高性能计算研究院（新加坡科研局下属机构））
FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Flower Labs（Flower实验室）；University of Cambridge（剑桥大学）；Zhejiang University（浙江大学）；Penn State University（宾夕法尼亚州立大学）；Alibaba Group（阿里巴巴集团）；University of Nevada, Reno（内华达大学拉斯维加斯分校）；Gachon University（加荣大学）；The University of Auckland（奥克兰大学）；The University of Melbourne（墨尔本大学）；The University of Adelaide（阿德莱德大学）；Sony AI（索尼人工智能）
Improving Local Training in Federated Learning via Temperature Scaling
School of Integrated Technology, Yonsei University（燕尼大学整合技术学院）；BK21 Graduate Program in Intelligent Semiconductor Technology, Yonsei University（燕尼大学智能半导体技术研究生项目）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）
In cross-device Federated Learning (FL), clients with low computational power train a common\linebreak[4] machine model by exchanging parameters via updates instead of potentially private data. Federated Dropout (FD) is a technique that improves the communication efficiency of a FL session by selecting a \emph{subset} of model parameters to be updated in each training round. However, compared to standard FL, FD produces considerably lower accuracy and faces a longer convergence time. In this paper, we leverage \textit{coding theory} to enhance FD by allowing different sub-models to be used at …
FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models Fine-Tuning
Beijing Key Laboratory of Data Intelligence and Security, National Engineering Research Center for Software Engineering, School of Computer Science, Peking University（北京数据智能与安全重点实验室、软件工程国家工程研究中心、计算机科学学院、北京大学）；Beijing Key Laboratory of Data Intelligence and Security, School of Software and Microelectronics, Peking University（北京数据智能与安全重点实验室、软件与微电子学院、北京大学）；Beijing Key Laboratory of Data Intelligence and Security, National Engineering Research Center for Software Engineering, Peking University（北京数据智能与安全重点实验室、软件工程国家工程研究中心、北京大学）
FedRef: Bayesian Fine-Tuning using a Reference Model to Mitigate Catastrophic Forgetting for Heterogeneous Federated Learning
School of Computer Science and Engineering（计算机科学与工程学院）；Center for Biosystems and Biotech Data Science（生物系统与生物技术数据科学中心）；Ghent University Global Campus, Incheon, Republic of Korea（因斯布鲁克大学全球校区，韩国）
Federated learning is a distributed, privacy-aware learning scenario which trains a single model on data belonging to several clients. Each client trains a local model on its data and the local models are then aggregated by a central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning is particularly important in these cases but traditional automated hyper-parameter tuning methods would require several training trials which is …
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
