FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering
\ School of Computing；Informatics University of Louisiana at Lafayette 1 Th rv \"a ld Circle Lafayette, LA USA；\ School of Computer Science；Technology East China Normal University 1 Th rv \"a ld Circle Shanghai China；\ Department of Computer；Information Sciences University of Delaware 1 Th rv \"a ld Circle Newark, DE USA；School of Computing；Informatics University of Louisiana at Lafayette；School of Computer Science；Technology East China Normal University；Department of Computer；Information Sciences University of Delaware
Federated learning (FL) is an emerging distributed machine learning paradigm that enables collaborative training of machine learning models over decentralized devices without exposing their local data. One of the major challenges in FL is the presence of uneven data distributions across client devices, violating the well-known assumption of independent-and-identically-distributed (IID) training samples in conventional machine learning. To address the performance degradation issue incurred by such data heterogeneity, clustered federated learning (CFL) shows its promise by grouping clients into separate learning clusters based on the similarity of their local data distributions. However, state-of-the-art CFL approaches require a large number of communication rounds to learn the distribution similarities during training until the formation of clusters is stabilized. Moreover, some of these algorithms heavily rely on a predefined number of clusters, thus limiting their flexibility and adaptability. In this paper, we propose {\em FedClust}, a novel approach for CFL that leverages the correlation between local model weights and the data distribution of clients. {\em FedClust} groups clients into clusters in a one-shot manner by measuring the similarity degrees among clients based on the strategically selected partial weights of locally trained models. We conduct extensive experiments on four benchmark datasets with different non-IID data settings. Experimental results demonstrate that {\em FedClust} achieves higher model accuracy up to $\sim$45\% as well as faster convergence with a significantly reduced communication cost up to 2.7$\times$ compared to its state-of-the-art counterparts.
FedClust: Optimizing Federated Learning on Non-IID Data through Weight-Driven Client Clustering
School of Computing and Informatics, University of Louisiana at Lafayette（路易斯安那大学拉法叶分校计算与信息学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；University of Delaware（特拉华大学）
Department of Engineering, University of Palermo, Palermo, Italy（工程系，帕尔梅罗大学）；Department of Computer Science, Missouri University of Science and Technology, Rolla, USA（计算机科学系，密苏里科技大学）；Department of Computer Science, BITS-Pilani, Dubai, UAE（计算机科学系，比斯帕利尼学院）
Tackling Noisy Clients in Federated Learning with End-to-end Label Correction
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
College of Computer Science and Technology（计算机科学与技术学院）；Zhejiang University（浙江大学）
Federated Learning (FL) is a promising distributed learning paradigm, which allows a number of data owners (also called clients) to collaboratively learn a shared model without disclosing each client's data. However, FL may fail to proceed properly, amid a state that we call negative federated learning (NFL). This paper addresses the problem of negative federated learning. We formulate a rigorous definition of NFL and analyze its essential cause. We propose a novel framework called LINDT for tackling NFL in run-time. The framework can potentially work with any neural-network-based FL systems f…
Tackling Privacy Heterogeneity in Differentially Private Federated Learning
Department of Information Engineering, The Chinese University of Hong Kong（信息工程系，香港中文大学）；School of Science and Engineering, Shenzhen Institute of Artificial Intelligence and Robotics for Society（科学与工程学院，深圳人工智能与机器人社会研究院）
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Yale University（耶鲁大学）
Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-…
Data-heterogeneous federated learning (FL) systems suffer from two significant sources of convergence error: 1) client drift error caused by performing multiple local optimization steps at clients, and 2) partial client participation error caused by the fact that only a small subset of the edge clients participate in every training round. We find that among these, only the former has received significant attention in the literature. To remedy this, we propose FedVARP, a novel variance reduction algorithm applied at the server that eliminates error due to partial client participation. To do so,…
The Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）
The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, which causes the client models drift further away from each other and degrades the aggregated global model's performance. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. Unfortunately, this reduces to regular training, which compromises clients' privacy and conflicts with the purpose of FL. In this paper, we put forth an idea to col…
Federated Learning is a fast growing area of ML where the training datasets are extremely distributed, all while dynamically changing over time. Models need to be trained on clients' devices without any guarantees for either homogeneity or stationarity of the local private data. The need for continual training has also risen, due to the ever-increasing production of in-task data. However, pursuing both directions at the same time is challenging, since client data privacy is a major constraint, especially for rehearsal methods. Herein, we propose a novel algorithm to address the incremental lea…
In federated optimization, heterogeneity in the clients' local datasets and computation speeds results in large variations in the number of local updates performed by each client in each communication round. Naive weighted aggregation of such models causes objective inconsistency, that is, the global model converges to a stationary point of a mismatched objective function which can be arbitrarily different from the true objective. This paper provides a general framework to analyze the convergence of federated heterogeneous optimization algorithms. It subsumes previously proposed methods such a…
We consider a standard federated learning (FL) architecture where a group of clients periodically coordinate with a central server to train a statistical model. We develop a general algorithmic framework called FedLin to tackle some of the key challenges intrinsic to FL, namely objective heterogeneity, systems heterogeneity, and infrequent and imprecise communication. Our framework is motivated by the observation that under these challenges, various existing FL algorithms suffer from a fundamental speed-accuracy conflict: they either guarantee linear convergence but to an incorrect point, or c…
University of Science and Technology of China (USTC)（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
Attributes skew hinders the current federated learning (FL) frameworks from consistent optimization directions among the clients, which inevitably leads to performance reduction and unstable convergence. The core problems lie in that: 1) Domain-specific attributes, which are non-causal and only locally valid, are indeliberately mixed into global aggregation. 2) The one-stage optimizations of entangled attributes cannot simultaneously satisfy two conflicting objectives, i.e., generalization and personalization. To cope with these, we proposed disentangled federated learning (DFL) to disentangle…
Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions
Department of Mathematical Sciences, Politecnico di Torino（数学科学系，都灵理工学院）；Istituto Nazionale di Fisica Nucleare, Sezione di Torino（国家核物理研究所，都灵分所）；Department of Electronics and Telecommunications, Politecnico di Torino（电子与电信系，都灵理工学院）
Tackling Non-IIDness in HAPS-Aided Federated Learning
Non-Terrestrial Networks (NTN) Lab, Department of Systems and Computer Engineering, Carleton University（卡尔顿大学系统与计算机工程系非地面网络实验室）；School of EECS, Ohio University（俄亥俄大学电气工程与计算机科学学院）
FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization
Hanoi University of Science and Technology（河内科学技术大学）；Nagoya University（名古屋大学）；Washington State University（华盛顿州立大学）；Swiss Federal Institute of Technology Lausanne（洛桑联邦理工学院）
FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来智能网络重点实验室）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；School of Engineering, Westlake University（西湖大学工程学院）
Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Zhongguancun Laboratory（中关村实验室）；Sichuan University（四川大学）；University at Buffalo（纽约州立大学布法罗分校）
University of California, Davis（加利福尼亚大学戴维斯分校）；Google（谷歌公司）
With more regulations tackling users' privacy-sensitive data protection in recent years, access to such data has become increasingly restricted and controversial. To exploit the wealth of data generated and located at distributed entities such as mobile phones, a revolutionary decentralized machine learning setting, known as Federated Learning, enables multiple clients located at different geographical locations to collaboratively learn a machine learning model while keeping all their data on-device. However, the scale and decentralization of federated learning present new challenges. Communic…
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have receiv…
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）
Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose $\texttt{FedCorr}$, a general multi-stage framework to…
KAUST（阿卜杜拉国王科技大学）；Samsung AI Center（三星人工智能中心）；University of Cambridge（剑桥大学）
Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform l…
Tackling Data Heterogeneity in Federated Learning via Loss Decomposition
School of Cyberspace, Hangzhou Dianzi University（杭州电子科技大学网络空间学院）；New H3C Technologies Co., Ltd.（新华三技术有限公司）；Computer Science and Engineering, University of California, Santa Cruz（加州大学圣克鲁兹分校计算机科学与工程系）；Department of Statistics and Actuarial Science, The University of Hong Kong（香港大学统计与精算科学系）
Protocol Choices Matter More Than the Aggregation Algorithm: Federated Learning for In-Hospital Mortality Prediction in Care-Unit and Multi-Hospital Federations
Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Norte (IFRN)（北大河州教育、科学和技术联邦学院（IFRN））
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that …
Huazhong University of Science and Technology（华中科技大学）；University of Waterloo（滑铁卢大学）；Northeastern University（东北大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
Traditional machine learning relies on a centralized data pipeline, i.e., data are provided to a central server for model training. In many applications, however, data are inherently fragmented. Such a decentralized nature of these databases presents the biggest challenge for collaboration: sending all decentralized datasets to a central server raises serious privacy concerns. Although there has been a joint effort in tackling such a critical issue by proposing privacy-preserving machine learning frameworks, such as federated learning, most state-of-the-art frameworks are built still in a cent…
In real-world federated learning scenarios, participants could have their own personalized labels which are incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains. However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same task from the same domain. In this work, to tackle these challenges, we introduce Factorized-FL, which allows to effectively tackle label-…
The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
Multiple medical institutions collaboratively training a model using federated learning (FL) has become a promising solution for maximizing the potential of data-driven models, yet the non-independent and identically distributed (non-iid) data in medical images is still an outstanding challenge in real-world practice. The feature heterogeneity caused by diverse scanners or protocols introduces a drift in the learning process, in both local (client) and global (server) optimizations, which harms the convergence as well as model performance. Many previous works have attempted to address the non-…
