Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment
College of Computing and Data Science, Nanyang Technological University, Singapore.（南洋理工大学计算机与数据科学学院）；Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada.（不列颠哥伦比亚大学电子与计算机工程系）；Vector Institute, Canada（加拿大向量研究所）；The Hong Kong Polytechnic University（香港理工大学）
Low-Rank Adaptation (LoRA) is widely used for federated fine-tuning. Yet under non-IID settings, it can substantially underperform full-parameter fine-tuning. Through with-high-probability robustness analysis, we uncover that this gap can be attributed to two coupled mismatches: (i) update-space mismatch, where clients optimize in a low-rank subspace but aggregation occurs in the full space; and (ii) optimizer-state mismatch, where unsynchronized adaptive states amplify drift across rounds. We propose FedGaLore, which combines client-side GaLore-style gradient-subspace optimization with server-side drift-robust synchronization of projected second-moment states via spectral shared-signal extraction, to address this challenge. Across NLU, vision, and NLG benchmarks, FedGaLore improves robustness and accuracy over state-of-the-art federated LoRA baselines in non-IID settings.
Though successful, federated learning presents new challenges for machine learning, especially when the issue of data heterogeneity, also known as Non-IID data, arises. To cope with the statistical heterogeneity, previous works incorporated a proximal term in local optimization or modified the model aggregation scheme at the server side or advocated clustered federated learning approaches where the central server groups agent population into clusters with jointly trainable data distributions to take the advantage of a certain level of personalization. While effective, they lack a deep elaborat…
Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
Institute of Engineering Research, Korea University（韩国大学工程研究所）；Kim Jaechul Graduate School of AI, KAIST（韩国科学技术院金在哲人工智能研究生院）；Industrial Management Engineering, Korea University（韩国大学产业管理工程学院）
Rethinking Client Drift in Federated Learning: A Logit Perspective
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究院）；Hubei Luojia Laboratory（湖北珞珈实验室）；Wuhan University（武汉大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；UC Santa Cruz（加州大学圣克鲁兹分校）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigoro…
Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（皇家理工学院电气工程与计算机科学学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）
University of Central Florida（中佛罗里达大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Tulane University（杜兰大学）
Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices). However, the data distribution among clients is often non-IID in nature, making efficient optimization difficult. To alleviate this issue, many FL algorithms focus on mitigating the effects of data heterogeneity across clients by introducing a variety of proximal terms, some incurring considerable compute and/or memory overheads, to restrain local updates with respect to the global model. Instead, we consider rethinking solutions to data hetero…
Institute of Computing Technology, Chinese Academy of Sciences, China（中国科学院计算技术研究所）；Purple Mountain Laboratories, China（紫金山实验室）；University of Chinese Academy of Sciences, China（中国科学院大学）
Federated Learning framework for LoRaWAN-enabled IIoT communication: A case study
Centre of Informatics and Systems of the University of Coimbra, CISUC（科英布拉大学信息与系统中心）；University of Coimbra（科英布拉大学）；Institute for Systems Engineering and Computers, INESC（系统工程与计算机研究所）；Sul-Rio-Grandense Federal Institute (IFSUL)（南里奥格兰德联邦学院）；Instituto de Telecomunicações（电信研究所）；Polytechnic Institute of Coimbra（科英布拉理工学院）；Coimbra Business School（科英布拉商学院）；INESC Coimbra（科英布拉INESC）
FedRIR: Rethinking Information Representation in Federated Learning
Institute for Clarity in Documentation（文献清晰研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗克伦古中心）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
Rethinking Federated Learning Over the Air: The Blessing of Scaling Up
Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学/伊利诺伊大学厄巴纳-香槟分校研究所）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Engineering Systems and Design, Singapore University of Technology and Design（新加坡科技与设计大学工程系统与设计系）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机科学学院）；Department of Computer and Information Science, Linköping University（_linköping大学计算机与信息科学系）
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo（人工智能商业创新中心，管理科学与系统系，布法罗大学）
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
School of Cyber Science and Technology Shenzhen Campus of Sun Yat-sen University（中山大学计算机科学与技术学院深圳校区）；School of Computer Science Faculty of Engineering The University of Sydney（悉尼大学计算机科学学院）；Generative AI Lab College of Computing and Data Science Nanyang Technological University（南洋理工大学计算与数据科学学院生成式人工智能实验室）
Rethinking Clustered Federated Learning in NOMA Enhanced Wireless Networks
School of Electrical and Electronic Engineering, The University of Manchester（曼彻斯特大学电气与电子工程学院）；Khalifa University（哈利法大学）；Department of Electrical Engineering and Computer Science, Khalifa University（哈利法大学电气工程与计算机科学系）；Department of Electrical and Electronic Engineering, University of Manchester（曼彻斯特大学电气与电子工程学院）
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
University of Manchester（曼彻斯特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；InstantX Team（InstantX团队）；University of Edinburgh（爱丁堡大学）；Aalto University（阿alto大学）
Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
HiThink Research（海思思考研究院）；Auburn University（奥本大学）；Kent State University（肯特州立大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Inria（法国国家信息与自动化研究所）；University of Montpellier（蒙彼利埃大学）；CNRS（法国国家科学研究中心）；LNCC（巴西国家计算科学实验室）；Fudan University（复旦大学）；BEDI Cloud（北电云）
FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Shanghai Jiao Tong University（上海交通大学）；Queen's University Belfast（女王大学贝尔法斯特分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Purdue University（普渡大学）；Rice University（里奇大学）
Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Con…
