Federated learning (FL), which has gained increasing attention recently, enables distributed devices to train a common machine learning (ML) model for intelligent inference cooperatively without data sharing.
However, problems in practical networks, such as non-independent-and-identically-distributed (non-iid) raw data and limited bandwidth, give rise to slow and unstable convergence of the FL training process.
To address these issues, we propose a new FL method that can significantly mitigate statistical heterogeneity through the depersonalization mechanism.
Particularly, we decouple the global and local optimization objectives by alternating stochastic gradient descent, thus reducing the accumulated variance in local update phases to accelerate the FL convergence.
Then we analyze the proposed method detailedly to show the proposed method converging at a sublinear speed in the general non-convex setting.
Finally, numerical results are conducted with experiments on public datasets to verify the effectiveness of our proposed method.
Zhejiang University（浙江大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
We develop a general framework unifying several gradient-based stochastic optimization methods for empirical risk minimization problems both in centralized and distributed scenarios. The framework hinges on the introduction of an augmented graph consisting of nodes modeling the samples and edges modeling both the inter-device communication and intra-device stochastic gradient computation. By designing properly the topology of the augmented graph, we are able to recover as special cases the renowned Local-SGD and DSGD algorithms, and provide a unified perspective for variance-reduction (VR) and…
Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions
Department of Mathematical Sciences, Politecnico di Torino（数学科学系，都灵理工学院）；Istituto Nazionale di Fisica Nucleare, Sezione di Torino（国家核物理研究所，都灵分所）；Department of Electronics and Telecommunications, Politecnico di Torino（电子与电信系，都灵理工学院）
Tackling Data Heterogeneity in Federated Learning via Loss Decomposition
School of Cyberspace, Hangzhou Dianzi University（杭州电子科技大学网络空间学院）；New H3C Technologies Co., Ltd.（新华三技术有限公司）；Computer Science and Engineering, University of California, Santa Cruz（加州大学圣克鲁兹分校计算机科学与工程系）；Department of Statistics and Actuarial Science, The University of Hong Kong（香港大学统计与精算科学系）
Tackling Privacy Heterogeneity in Differentially Private Federated Learning
Department of Information Engineering, The Chinese University of Hong Kong（信息工程系，香港中文大学）；School of Science and Engineering, Shenzhen Institute of Artificial Intelligence and Robotics for Society（科学与工程学院，深圳人工智能与机器人社会研究院）
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；UC Santa Cruz（加州大学圣克鲁兹分校）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigoro…
We consider a standard federated learning (FL) architecture where a group of clients periodically coordinate with a central server to train a statistical model. We develop a general algorithmic framework called FedLin to tackle some of the key challenges intrinsic to FL, namely objective heterogeneity, systems heterogeneity, and infrequent and imprecise communication. Our framework is motivated by the observation that under these challenges, various existing FL algorithms suffer from a fundamental speed-accuracy conflict: they either guarantee linear convergence but to an incorrect point, or c…
Stanford University（斯坦福大学）；The University of Hong Kong（香港大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
The collection and curation of large-scale medical datasets from multiple institutions is essential for training accurate deep learning models, but privacy concerns often hinder data sharing. Federated learning (FL) is a promising solution that enables privacy-preserving collaborative learning among different institutions, but it generally suffers from performance deterioration due to heterogeneous data distributions and a lack of quality labeled data. In this paper, we present a robust and label-efficient self-supervised FL framework for medical image analysis. Our method introduces a novel T…
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Yale University（耶鲁大学）
Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-…
The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）
The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, which causes the client models drift further away from each other and degrades the aggregated global model's performance. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. Unfortunately, this reduces to regular training, which compromises clients' privacy and conflicts with the purpose of FL. In this paper, we put forth an idea to col…
Tackling Data Heterogeneity in Federated Time Series Forecasting
The University of Queensland（昆士兰大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；City University of Hong Kong（香港城市大学）；Griffith University（格里菲斯大学）；Institute of Science Tokyo（东京科学大学）
FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering
\ School of Computing；Informatics University of Louisiana at Lafayette 1 Th rv \"a ld Circle Lafayette, LA USA；\ School of Computer Science；Technology East China Normal University 1 Th rv \"a ld Circle Shanghai China；\ Department of Computer；Information Sciences University of Delaware 1 Th rv \"a ld Circle Newark, DE USA；School of Computing；Informatics University of Louisiana at Lafayette；School of Computer Science；Technology East China Normal University；Department of Computer；Information Sciences University of Delaware
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such a…
FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization
Hanoi University of Science and Technology（河内科学技术大学）；Nagoya University（名古屋大学）；Washington State University（华盛顿州立大学）；Swiss Federal Institute of Technology Lausanne（洛桑联邦理工学院）
Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computing & Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Cyber Science and Technology, Sun Yat-sen University（中山大学网络空间科学与技术学院）；University of Maryland, College Park（马里兰大学帕克分校）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
Efficient Conformal Prediction under Data Heterogeneity
Ecole Polytechnique（巴黎综合理工学院）；Lagrange Mathematics and Computing Research Center（拉格朗日数学与计算研究中心）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；HSE University（高等经济大学）；Technology Innovation Institute（技术创新研究院）
Tackling Non-IIDness in HAPS-Aided Federated Learning
Non-Terrestrial Networks (NTN) Lab, Department of Systems and Computer Engineering, Carleton University（卡尔顿大学系统与计算机工程系非地面网络实验室）；School of EECS, Ohio University（俄亥俄大学电气工程与计算机科学学院）
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have receiv…
Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Zhongguancun Laboratory（中关村实验室）；Sichuan University（四川大学）；University at Buffalo（纽约州立大学布法罗分校）
Viterbi School of Engineering（维特比工程学院）；University of Southern California（南加利福尼亚大学）
Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy. An essential but rarely studied challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL compared to centralized training due to the fact that FL users are often reluctant to label their private data. Furthermore, due to the heterogeneous nature of the data at edge devices, it is crucial to develop personalized models. In this paper we propose self-supervised federated learni…
KAUST（阿卜杜拉国王科技大学）；Samsung AI Center（三星人工智能中心）；University of Cambridge（剑桥大学）
Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform l…
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that …
Rutgers University（罗格斯大学）；US Army Research Lab（美国陆军研究实验室）
We consider the problem of a Parameter Server (PS) that wishes to learn a model that fits data distributed on the nodes of a graph. We focus on Federated Learning (FL) as a canonical application. One of the main challenges of FL is the communication bottleneck between the nodes and the parameter server. A popular solution in the literature is to allow each node to do several local updates on the model in each iteration before sending it back to the PS. While this mitigates the communication bottleneck, the statistical heterogeneity of the data owned by the different nodes has proven to delay c…
North Carolina State University（北卡罗来纳州立大学）；Cisco Research（思科研究院）；Cisco Systems, Inc.（思科系统公司）；The George Washington University（乔治·华盛顿大学）；University of Electro-Communications（东京电机大学）；Nagoya University（名古屋大学）
The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
Multiple medical institutions collaboratively training a model using federated learning (FL) has become a promising solution for maximizing the potential of data-driven models, yet the non-independent and identically distributed (non-iid) data in medical images is still an outstanding challenge in real-world practice. The feature heterogeneity caused by diverse scanners or protocols introduces a drift in the learning process, in both local (client) and global (server) optimizations, which harms the convergence as well as model performance. Many previous works have attempted to address the non-…
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）
Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose $\texttt{FedCorr}$, a general multi-stage framework to…
