Federated learning (FL) provides a privacy-preserving solution for fine-tuning pre-trained large language models (LLMs) using distributed private datasets, enabling task-specific adaptation while preserving data privacy. However, fine-tuning the extensive parameters in LLMs is particularly challenging in resource-constrained federated scenarios due to the significant communication and computational costs. To gain a deeper understanding of how these challenges can be addressed, this article conducts a comparative analysis three advanced federated LLM (FedLLM) frameworks that integrate knowledge distillation (KD) and split learning (SL) to mitigate these issues: 1) FedLLMs, where clients upload model parameters or gradients to enable straightforward and effective fine-tuning; 2) KD-FedLLMs, which leverage KD for efficient knowledge sharing via logits; and 3) Split-FedLLMs, which split the LLMs into two parts, with one part executed on the client and the other one on the server, to balance the computational load. Each framework is evaluated based on key performance metrics, including model accuracy, communication overhead, and client-side computational load, offering insights into their effectiveness for various federated fine-tuning scenarios. Through this analysis, we identify framework-specific optimization opportunities to enhance the efficiency of FedLLMs and discuss broader research directions, highlighting open opportunities to better adapt FedLLMs for real-world applications. A use case is presented to demonstrate the performance comparison of these three frameworks under varying configurations and settings.
ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada（电气与计算机工程系，不列颠哥伦比亚大学）；Department of Computer Science and Engineering, Southern University of Science and Technology, China（计算机科学与工程系，南方科技大学）
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
The Hong Kong Polytechnic University（香港理工大学）；OceanBase, Ant Group（蚂蚁集团OceanBase）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Mathematics and Statistics, Xidian University（西安电子科技大学数学与统计学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, China University of Geosciences（中国地质大学计算机学院）
Federated Customization of Large Models: Approaches, Experiments, and Insights
College of Physics and Information Engineering, Fuzhou University（物理与信息工程学院，福州大学）；Data61, CSIRO（Data61，CSIRO）；Department of Computer Science and Information Technology, La Trobe University（计算机科学与信息科技系，拉筹夫大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMs
University of California, Irvine（加州大学欧文分校）；University of Southern California（南加州大学）；TensorOpera Inc.（TensorOpera公司）；Lehigh University（理海大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Texas A&M University（德克萨斯农工大学）；Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；USC（南加州大学）
Cross-Cloud Data Privacy Protection: Optimizing Collaborative Mechanisms of AI Systems by Integrating Federated Learning and LLMs
College of Computing and Information Science（计算与信息科学学院）；Siebel School of Computing and Data Science（计算与数据科学学院）；Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；New York, USA（美国纽约）；Champaign, Illinois, USA（美国伊利诺伊州香槟）
Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data
Washington State University（华盛顿州立大学）；Princeton University（普林斯顿大学）；IBM Research（IBM研究院）；University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣迭戈分校）
School of Advanced Interdisciplinary Sciences, UCAS（中国科学院大学先进交叉学科学院）；MAIS, CASIA（中国科学院自动化所人工智能研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港科技大学人工智能与机器人中心，中国科学院）
Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a nove…
We describe the design of our federated task processing system. Originally, the system was created to support two specific federated tasks: evaluation and tuning of on-device ML systems, primarily for the purpose of personalizing these systems. In recent years, support for an additional federated task has been added: federated learning (FL) of deep neural networks. To our knowledge, only one other system has been described in literature that supports FL at scale. We include comparisons to that system to help discuss design decisions and attached trade-offs. Finally, we describe two specific la…
