Federated learning (FL) collaboratively models user data in a decentralized way. However, in the real world, non-identical and independent data distributions (non-IID) among clients hinder the performance of FL due to three issues, i.e., (1) the class statistics shifting, (2) the insufficient hierarchical information utilization, and (3) the inconsistency in aggregating clients. To address the above issues, we propose HyperFed which contains three main modules, i.e., hyperbolic prototype Tammes initialization (HPTI), hyperbolic prototype learning (HPL), and consistent aggregation (CA). Firstly, HPTI in the server constructs uniformly distributed and fixed class prototypes, and shares them with clients to match class statistics, further guiding consistent feature representation for local clients. Secondly, HPL in each client captures the hierarchical information in local data with the supervision of shared class prototypes in the hyperbolic model space. Additionally, CA in the server mitigates the impact of the inconsistent deviations from clients to server. Extensive studies of four datasets prove that HyperFed is effective in enhancing the performance of FL under the non-IID set.
Pennsylvania State University（宾夕法尼亚州立大学）；IBM Research - Almaden（IBM阿尔马登研究中心）
Federated Learning (FL) is a paradigm for jointly training machine learning algorithms in a decentralized manner which allows for parties to communicate with an aggregator to create and train a model, without exposing the underlying raw data distribution of the local parties involved in the training process. Most research in FL has been focused on Neural Network-based approaches, however Tree-Based methods, such as XGBoost, have been underexplored in Federated Learning due to the challenges in overcoming the iterative and additive characteristics of the algorithm. Decision tree-based models, i…
National University of Defense Technology（国防科技大学）；IHPC, ASTAR（新加坡科技研究局高性能计算研究所）；University of Macau（澳门大学）
Federated learning (FL) allows multiple clients to collectively train a high-performance global model without sharing their private data. However, the key challenge in federated learning is that the clients have significant statistical heterogeneity among their local data distributions, which would cause inconsistent optimized local models on the client-side. To address this fundamental dilemma, we propose a novel federated learning algorithm with local drift decoupling and correction (FedDC). Our FedDC only introduces lightweight modifications in the local training phase, in which each client…
College of Computing and Informatics, Sungkyunkwan University（成均馆大学计算与信息学院）；Bio-Medical IT Convergence Research Division, Electronics and Telecommunications Research Institute（电子通信研究院生物医学IT融合研究部）；Samsung Medical Center（三星医疗中心）
While machine learning techniques are being applied to various fields for their exceptional ability to find complex relations in large datasets, the strengthening of regulations on data ownership and privacy is causing increasing difficulty in its application to medical data. In light of this, Federated Learning has recently been proposed as a solution to train on private data without breach of confidentiality. This conservation of privacy is particularly appealing in the field of healthcare, where patient data is highly confidential. However, many studies have shown that its assumption of Ind…
Dataset Distillation-based Hybrid Federated Learning on Non-IID Data
College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China（浙江工业大学信息工程学院，杭州 310023，中国）；Zhejiang Key Laboratory of Intelligent Perception and Control for Complex Systems, Hangzhou, China（浙江省复杂系统智能感知与控制重点实验室，杭州，中国）；College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China（浙江大学控制科学与工程学院，杭州 310027，中国）；Computing Science and Internet of Things, Newcastle University, NE1 7RU Newcastle, UK（新castle大学计算科学与物联网学院，NE1 7RU Newcastle, UK）
University of Maryland（马里兰大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Numerous large-scale chest x-ray datasets have spearheaded expert-level detection of abnormalities using deep learning. However, these datasets focus on detecting a subset of disease labels that could be present, thus making them distributed and non-iid with partial labels. Recent literature has indicated the impact of batch normalization layers on the convergence of federated learning due to domain shift associated with non-iid data with partial labels. To that end, we propose FedFBN, a federated learning framework that draws inspiration from transfer learning by using pretrained networks as …
Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data
Washington State University（华盛顿州立大学）；Princeton University（普林斯顿大学）；IBM Research（IBM研究院）；University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣迭戈分校）
Xiamen University（厦门大学）；Hong Kong Baptist University（香港浸会大学）
Federated learning provides a privacy guarantee for generating good deep learning models on distributed clients with different kinds of data. Nevertheless, dealing with non-IID data is one of the most challenging problems for federated learning. Researchers have proposed a variety of methods to eliminate the negative influence of non-IIDness. However, they only focus on the non-IID data provided that the universal class distribution is balanced. In many real-world applications, the universal class distribution is long-tailed, which causes the model seriously biased. Therefore, this paper studi…
Technical University of Munich（慕尼黑工业大学）；University of Hamburg（汉堡大学）；Friedrich-Alexander University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Federated learning has faced performance and network communication challenges, especially in the environments where the data is not independent and identically distributed (IID) across the clients. To address the former challenge, we introduce the federated-centralized concordance property and show that the federated single-mini-batch training approach can achieve comparable performance as the corresponding centralized training in the Non-IID environments. To deal with the latter, we present the federated multi-mini-batch approach and illustrate that it can establish a trade-off between the pe…
Federated Learning (FL) is a distributed learning paradigm that enables a large number of resource-limited nodes to collaboratively train a model without data sharing. The non-independent-and-identically-distributed (non-i.i.d.) data samples invoke discrepancies between the global and local objectives, making the FL model slow to converge. In this paper, we proposed Optimal Aggregation algorithm for better aggregation, which finds out the optimal subset of local updates of participating nodes in each global round, by identifying and excluding the adverse local updates via checking the relation…
TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency
School of Computing, Southern Illinois University（南伊利诺伊大学计算机学院）；Security, Privacy and Intelligence for Edge Devices Laboratory (SPEED Lab)（边缘设备安全、隐私与智能实验室（SPEED Lab））
Warsaw University of Technology（华沙理工大学）；Systems Research Institute Polish Academy of Sciences（波兰科学院系统研究所）
One of the important problems in federated learning is how to deal with unbalanced data. This contribution introduces a novel technique designed to deal with label skewed non-IID data, using adversarial inputs, created by the I-FGSM method. Adversarial inputs guide the training process and allow the Weighted Federated Averaging to give more importance to clients with 'selected' local label distributions. Experimental results, gathered from image classification tasks, for MNIST and CIFAR-10 datasets, are reported and analyzed.
Delft University of Technology（代尔夫特理工大学）；University of Greenwich（格林威治大学）
Federated learning is an emerging concept in the domain of distributed machine learning. This concept has enabled GANs to benefit from the rich distributed training data while preserving privacy. However, in a non-iid setting, current federated GAN architectures are unstable, struggling to learn the distinct features and vulnerable to mode collapse. In this paper, we propose a novel architecture MULTI-FLGAN to solve the problem of low-quality images, mode collapse and instability for non-iid datasets. Our results show that MULTI-FLGAN is four times as stable and performant (i.e. high inception…
Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences and the University of Chinese Academy of Sciences（北京移动计算与 pervasive 设备重点实验室，计算技术研究所，中国科学院和中国科学院大学）；Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences（北京移动计算与 pervasive 设备重点实验室，计算技术研究所，中国科学院）；Shandong Academy of Intelligent Computing Technology（山东省智能计算技术研究院）；Frontier Sciences and Education, Chinese Academy of Sciences（前沿科学与教育，中国科学院）
Federated learning is an emerging distributed machine learning framework for privacy preservation. However, models trained in federated learning usually have worse performance than those trained in the standard centralized learning mode, especially when the training data are not independent and identically distributed (Non-IID) on the local devices. In this survey, we pro-vide a detailed analysis of the influence of Non-IID data on both parametric and non-parametric machine learning models in both horizontal and vertical federated learning. In addition, cur-rent research work on handling chall…
Generalizable Hyperparameter Optimization for Federated Learning on Non-IID Cancer Images
Institute of Exact and Technological Sciences, Federal University of Viçosa - UFV, Rio Paranaíba-MG, Brazil（精确与技术科学研究所，弗拉维亚联邦大学-UFV，里奥帕拉纳伊巴-巴西）；Department of Computing, Federal University of São Carlos, São Carlos-SP, Brazil（计算系，圣卡洛斯联邦大学，圣卡洛斯-巴西）
University of Technology Sydney（悉尼科技大学）；Monash University（莫纳什大学）；Data61, CSIRO（联邦科学与工业研究组织Data61研究所）
Graph neural networks (GNNs) have shown their superiority in modeling graph data. Owing to the advantages of federated learning, federated graph learning (FGL) enables clients to train strong GNN models in a distributed manner without sharing their private data. A core challenge in federated systems is the non-IID problem, which also widely exists in real-world graph data. For example, local data of clients may come from diverse datasets or even domains, e.g., social networks and molecules, increasing the difficulty for FGL methods to capture commonly shared knowledge and learn a generalized e…
McGill University（麦吉尔大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos. Though successfully possessing advantages in both scale and privacy, federated learning is hurt by domain shift problems, where the learning models are unable to generalize to unseen domains whose data distribution is non-i.i.d. with respect to the training domains. In this study, we propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages th…
FedCoSR: Personalized Federated Learning with Contrastive Shareable Representations for Label Heterogeneity in Non-IID Data
Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University（数据科学与人工智能系，IT学院和莫纳什能源研究所，莫纳什大学）；School of Computer Science and Technology, University of Electronic Science and Technology of China (UESTC), Chengdu, and Shenzhen Institute for Advanced Study of UESTC, Shenzhen, China（计算机科学与技术学院，电子科技大学（UESTC）成都校区，以及UESTC深圳高级研究所，深圳，中国）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Federated learning has emerged as a new paradigm of collaborative machine learning; however, it has also faced several challenges such as non-independent and identically distributed(IID) data and high communication cost. To this end, we propose a novel framework of IID and communication-aware group federated learning that simultaneously maximizes both accuracy and communication speed by grouping nodes based on data distributions and physical locations of the nodes. Furthermore, we provide a formal convergence analysis and an efficient optimization algorithm called FedAvg-IC. Experimental resul…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Korea University（高丽大学）
Federated Learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning however comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods however are only of limited utility in the…
Federated learning refers to the task of machine learning based on decentralized data from multiple clients with secured data privacy. Recent studies show that quantum algorithms can be exploited to boost its performance. However, when the clients' data are not independent and identically distributed (IID), the performance of conventional federated algorithms is known to deteriorate. In this work, we explore the non-IID issue in quantum federated learning with both theoretical and numerical analysis. We further prove that a global quantum channel can be exactly decomposed into local channels t…
Shell Global Solutions International B.V.（壳牌全球解决方案国际有限公司）；Delft University of Technology (TU Delft)（代尔夫特理工大学）；Myant Inc.（迈安特公司）
Classical federated learning approaches incur significant performance degradation in the presence of non-IID client data. A possible direction to address this issue is forming clusters of clients with roughly IID data. Most solutions following this direction are iterative and relatively slow, also prone to convergence issues in discovering underlying cluster formations. We introduce federated learning with taskonomy (FLT) that generalizes this direction by learning the task-relatedness between clients for more efficient federated aggregation of heterogeneous data. In a one-off process, the ser…
Key Laboratory of Embedded System and Service Computing (Tongji University), Ministry of Education（嵌入式系统与服务计算教育部重点实验室（同济大学））
Federated learning is a novel framework that enables resource-constrained edge devices to jointly learn a model, which solves the problem of data protection and data islands. However, standard federated learning is vulnerable to Byzantine attacks, which will cause the global model to be manipulated by the attacker or fail to converge. On non-iid data, the current methods are not effective in defensing against Byzantine attacks. In this paper, we propose a Byzantine-robust framework for federated learning via credibility assessment on non-iid data (BRCA). Credibility assessment is designed to d…
Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to 55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class…
Federated learning (FL) is an emerging technique used to collaboratively train a global machine learning model while keeping the data localized on the user devices. The main obstacle to FL's practical implementation is the Non-Independent and Identical (Non-IID) data distribution across users, which slows convergence and degrades performance. To tackle this fundamental issue, we propose a method (ComFed) that enhances the whole training process on both the client and server sides. The key idea of ComFed is to simultaneously utilize client-variance reduction techniques to facilitate server aggr…
East China Normal University（华东师范大学）；Zaozhuang University（枣庄学院）；University of Houston（休斯顿大学）
As a promising method of central model training on decentralized device data while securing user privacy, Federated Learning (FL)is becoming popular in Internet of Things (IoT) design. However, when the data collected by IoT devices are highly skewed in a non-independent and identically distributed (non-IID) manner, the accuracy of vanilla FL method cannot be guaranteed. Although there exist various solutions that try to address the bottleneck of FL with non-IID data, most of them suffer from extra intolerable communication overhead and low model accuracy. To enable fast and accurate FL, this …
As a mechanism for devices to update a global model without sharing data, federated learning bridges the tension between the need for data and respect for privacy. However, classic FL methods like Federated Averaging struggle with non-iid data, a prevalent situation in the real world. Previous solutions are sub-optimal as they either employ a small shared global subset of data or greater number of models with increased communication costs. We propose FedFMC (Fork-Merge-Consolidate), a method that dynamically forks devices into updating different global models then merges and consolidates separ…
Federated learning allows multiple clients to collaborate to train high-performance deep learning models while keeping the training data locally. However, when the local data of all clients are not independent and identically distributed (i.e., non-IID), it is challenging to implement this form of efficient collaborative learning. Although significant efforts have been dedicated to addressing this challenge, the effect on the image classification task is still not satisfactory. In this paper, we propose FedProc: prototypical contrastive federated learning, which is a simple and effective feder…
In Federated Learning (FL), models are as fragile as centrally trained models against adversarial examples. However, the adversarial robustness of federated learning remains largely unexplored. This paper casts light on the challenge of adversarial robustness of federated learning. To facilitate a better understanding of the adversarial vulnerability of the existing FL methods, we conduct comprehensive robustness evaluations on various attacks and adversarial training methods. Moreover, we reveal the negative impacts induced by directly adopting adversarial training in FL, which seriously hurt…
The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
Federated learning (FL) has recently emerged as a popular privacy-preserving collaborative learning paradigm. However, it suffers from the non-independent and identically distributed (non-IID) data among clients. In this paper, we propose a novel framework, named Synthetic Data Aided Federated Learning (SDA-FL), to resolve this non-IID challenge by sharing synthetic data. Specifically, each client pretrains a local generative adversarial network (GAN) to generate differentially private synthetic data, which are uploaded to the parameter server (PS) to construct a global shared synthetic datase…
