In this paper, we question the rationale behind propagating large numbers of parameters through a distributed system during federated learning. We start by examining the rank characteristics of the subspace spanned by gradients across epochs (i.e., the gradient-space) in centralized model training, and observe that this gradient-space often consists of a few leading principal components accounting for an overwhelming majority (95-99%) of the explained variance. Motivated by this, we propose the "Look-back Gradient Multiplier" (LBGM) algorithm, which exploits this low-rank property to enable gradient recycling between model update rounds of federated learning, reducing transmissions of large parameters to single scalars for aggregation. We analytically characterize the convergence behavior of LBGM, revealing the nature of the trade-off between communication savings and model performance. Our subsequent experimental results demonstrate the improvement LBGM obtains in communication overhead compared to conventional federated learning on several datasets and deep learning models. Additionally, we show that LBGM is a general plug-and-play algorithm that can be used standalone or stacked on top of existing sparsification techniques for distributed model training.
Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment
College of Computing and Data Science, Nanyang Technological University, Singapore.（南洋理工大学计算机与数据科学学院）；Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada.（不列颠哥伦比亚大学电子与计算机工程系）；Vector Institute, Canada（加拿大向量研究所）；The Hong Kong Polytechnic University（香港理工大学）
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decoupling
School of Computer Science, Wuhan University（武汉大学计算机学院）；Faculty of Artificial Intelligence in Education, Central China Normal University（中央财经大学教育人工智能学院）
FedSteer: Taming Extreme Gradient Staleness in Federated Learning with Corrective Projections and Caching
Carnegie Mellon University（卡内基梅隆大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Avignon（阿维尼昂大学）；Singapore University of Technology and Design（新加坡科技与设计大学）；University of Washington（华盛顿大学）
FedCova: Robust Federated Covariance Learning Against Noisy Labels
Department of Information Engineering（信息工程系）；The Chinese University of Hong Kong（香港中文大学）；National Key Laboratory on Wireless Communications（无线通信国家重点实验室）；University of Electronic Science and Technology of China（电子科学与技术大学）
X-VFL: A New Vertical Federated Learning Framework with Cross Completion and Decision Subspace Alignment
Singapore Management University & University of Pennsylvania（新加坡国立管理学院及宾夕法尼亚大学）；Singapore Management University & Beijing Jiaotong University（新加坡国立管理学院及北京交通大学）；Singapore Management University（新加坡国立管理学院）
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University（虚拟现实技术与系统国家重点实验室，计算机科学与工程学院，北京航空航天大学）；Zhongguancun Laboratory（中关村实验室）；Center for AI Business Innovation, Department of Management Science and Systems, School of Management, University at Buffalo（人工智能商业创新中心，管理科学与系统系，布法罗大学）
UMEDA: Unified Multi-modal Efficient Data Fusion for Privacy-Preserving Graph Federated Learning via Spectral-Gated Attention and Diffusion-Based Operator Alignment
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，Location，Country）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，Location，Country）
Department of Biomedical Engineering, Linköping University, Sweden（_linköping大学生物医学工程系）；Department of Computer and Information Science, Linköping University, Sweden（_linköping大学计算机与信息科学系）；Center for Medical Image Science and Visualization (CMIV), Linköping University, Sweden（_linköping大学医学影像科学与可视化中心）
Towards Federated Domain Unlearning: Verification Methodologies and Challenges
University of Macau（澳门大学）；University Of Electronic Science And Technology（电子科技大学）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究院）
School of Advanced Interdisciplinary Sciences, UCAS（中国科学院大学先进交叉学科学院）；MAIS, CASIA（中国科学院自动化所人工智能研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港科技大学人工智能与机器人中心，中国科学院）
Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statistics
Department of Sciences and Methods for Engineering（工程科学与方法系）；Artificial Intelligence Research and Innovation Center（人工智能研究与创新中心）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）
School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；School of Software, Dalian University of Technology（大连理工大学软件学院）；School of Telecommunications Engineering, Xidian University（西安电子科技大学电信工程学院）
FedSurrogate: Backdoor Defense in Federated Learning via Layer Criticality and Surrogate Replacement
The University of Manchester, UK（英国曼彻斯特大学）；Institute for Infocomm Research, A*STAR, Singapore（新加坡信息通信研究所）；Monash University, Australia（澳大利亚墨尔本大学）；COSIC, KU Leuven, Belgium（比利时列日大学COSIC实验室）
Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach
Colleague of Computer Science and Technology, National University of Defense Technology, Changsha, China（计算机科学与技术系，国防科技大学，中国长沙）；City University of Hong Kong, Hong Kong, China（香港城市大学，中国香港）；Shenzhen University, Shenzhen, China（深圳大学，中国深圳）
