Fine-tuning is essential to adapt general-purpose large language models (LLMs) to domain-specific tasks. As a privacy-preserving framework to leverage decentralized data for collaborative model training, Federated Learning (FL) is gaining popularity in LLM fine-tuning, but remains challenging due to the high cost of transmitting full model parameters and computing full gradients on resource-constrained clients. While Parameter-Efficient Fine-Tuning (PEFT) methods are widely used in FL to reduce communication and memory costs, they often sacrifice model performance compared to FFT. This paper proposes FedKRSO (Federated $K$-Seed Random Subspace Optimization), a novel method that enables communication and memory efficient FFT of LLMs in federated settings. In FedKRSO, clients update the model within a shared set of random low-dimension subspaces generated by the server to save memory usage. Furthermore, instead of transmitting full model parameters in each FL round, clients send only the model update accumulators along the subspaces to the server, enabling efficient global model aggregation and dissemination. By using these strategies, FedKRSO can substantially reduce communication and memory overhead while overcoming the performance limitations of PEFT, closely approximating the performance of federated FFT. The convergence properties of FedKRSO are analyzed rigorously under general FL settings. Extensive experiments on the GLUE benchmark across diverse FL scenarios demonstrate that FedKRSO achieves both superior performance and low communication and memory overhead, paving the way towards on federated LLM fine-tuning at the resource-constrained edge.
ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada（电气与计算机工程系，不列颠哥伦比亚大学）；Department of Computer Science and Engineering, Southern University of Science and Technology, China（计算机科学与工程系，南方科技大学）
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing
Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
Department of Computational and Data Sciences, Indian Institute of Science（计算与数据科学系，印度科学研究院）；Accenture, Japan（日本Accenture公司）；Google, India（印度Google公司）；Indian Institute of Science（印度科学研究院）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
Department of Electrical and Computer Engineering, and the Institute of New Media and Communications, Seoul National University (SNU)（电气与计算机工程系，以及新媒体与通讯研究所，首尔国立大学）
Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州市先进研究 institute，中国科学技术大学）；School of Computer Science and Technology, China University of Mining and Technology（计算机科学与技术学院，中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education（教育部矿山数字化工程研究中心）
