Large Language Models (LLMs) have become indispensable in numerous real-world applications. However, fine-tuning these models at scale, especially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing approaches often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typically comes at the cost of model accuracy. To this end, we propose federated full-parameter tuning at scale for LLMs (Ferret), the first first-order method with shared randomness to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. Ferret accomplishes this through three aspects: (i) it employs widely used first-order methods for efficient local updates; (ii) it projects these updates into a low-dimensional space to considerably reduce communication overhead; and (iii) it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that Ferret significantly enhances the scalability of existing federated full-parameter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accuracy. Our implementation is available at https://github.com/allen4747/Ferret.
ZorBA: Zeroth-order Federated Fine-tuning of LLMs with Heterogeneous Block Activation
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, Canada（电气与计算机工程系，不列颠哥伦比亚大学）；Department of Computer Science and Engineering, Southern University of Science and Technology, China（计算机科学与工程系，南方科技大学）
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing
Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers
Department of Computational and Data Sciences, Indian Institute of Science（计算与数据科学系，印度科学研究院）；Accenture, Japan（日本Accenture公司）；Google, India（印度Google公司）；Indian Institute of Science（印度科学研究院）
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
Department of Electrical and Computer Engineering, and the Institute of New Media and Communications, Seoul National University (SNU)（电气与计算机工程系，以及新媒体与通讯研究所，首尔国立大学）
Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州市先进研究 institute，中国科学技术大学）；School of Computer Science and Technology, China University of Mining and Technology（计算机科学与技术学院，中国矿业大学）；Mine Digitization Engineering Research Center of the Ministry of Education（教育部矿山数字化工程研究中心）
