FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Shanghai Jiao Tong University（上海交通大学）；Queen's University Belfast（女王大学贝尔法斯特分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Purdue University（普渡大学）；Rice University（里奇大学）
Peishen Yan, Yang Hua, Hao Wang, Jiaru Zhang, Xiaoyu Wu, Tao Song, Haibing Guan
Federated fine-tuning of large language models (LLMs) with low-rank adaptation (LoRA) offers a communication-efficient and privacy-preserving solution for task-specific adaptation. Naive aggregation of LoRA modules introduces noise due to mathematical incorrectness when averaging the downsampling and upsampling matrices independently. However, existing noise-free aggregation strategies inevitably compromise the structural expressiveness of LoRA, limiting its ability to retain client-specific adaptations by either improperly reconstructing the low-rank structure or excluding partially trainable components. We identify this problem as loss of training momentum, where LoRA updates fail to accumulate effectively across rounds, resulting in slower convergence and suboptimal performance. To address this, we propose FedMomentum, a novel framework that enables structured and momentum-preserving LoRA aggregation via singular value decomposition (SVD). Specifically, after aggregating low-rank updates in a mathematically correct manner, FedMomentum applies SVD to extract the dominant components that capture the main update directions. These components are used to reconstruct the LoRA modules with the same rank, while residual components can be retained and later merged into the backbone to preserve semantic information and ensure robustness. Extensive experiments across multiple tasks demonstrate that FedMomentum consistently outperforms prior state-of-the-art methods in convergence speed and final accuracy.
Federated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray Cohorts
North Carolina A&T State University（北卡罗来纳农工州立大学）；Tribhuvan University（特里布文大学）；University of Tennessee-Knoxville（田纳西大学诺克斯维尔分校）；University of South Dakota（南达科他大学）
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Beijing Laboratory of Advanced Information Network（北京先进信息网络实验室）；Beijing Key Laboratory of Network System Architecture and Convergence（北京网络系统架构与融合重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Department of Electrical and Computer Engineering and Institute for Data Science and Computing（电气与计算机工程系和数据科学与计算研究所）；University of Miami（迈阿密大学）；Information Systems Technology and Design Pillar（信息系统技术与设计支柱）
Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling
National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications（移动网络技术国家工程研究中心，北京邮电大学）；Hong Kong Baptist University（香港 Baptist大学）；Peng Cheng Laboratory（鹏城实验室）
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke University（杜克大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
Department of Electrical and Computer Engineering, and the Institute of New Media and Communications, Seoul National University (SNU)（电气与计算机工程系，以及新媒体与通讯研究所，首尔国立大学）
WinFLoRA: Incentivizing Client-Adaptive Aggregation in Federated LoRA under Privacy Heterogeneity
RMIT University（皇家墨尔本理工大学）；Huazhong University of Science and Technology（华中科技大学）；CSIRO’s Data61 and Responsible AI Research (RAIR) Centre（澳大利亚CSIRO数据61和负责任的人工智能研究（RAIR）中心）
Selective Aggregation for Low-Rank Adaptation in Federated Learning
The University of Hong Kong（香港大学）；Stanford University（斯坦福大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Materials Innovation Institute for Life Sciences and Energy (MILES), HKU-SIRI（香港大学-SIRI生命科学与能源材料创新研究所）
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai（人工智能与未来网络研究院，北京师范大学，珠海）；School of Computer Science and Artificial Intelligence, Zhengzhou University（计算机科学与人工智能学院，郑州大学）；School of Software, Nanchang University（软件学院，南昌大学）
HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization
Department of Engineering, King’s College London（伦敦国王学院工程系）；Advanced Technology Group, Ericsson Inc.（爱立信公司先进科技组）；Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学数字通信研究所）
Department of Electronic Engineering（电子工程系）；Fudan University（复旦大学）；School of Computing（计算学院）；Macquarie University（麦觉瑞大学）；Department of Communication Science and Engineering（通信科学与工程系）
