Federated learning (FL) is a promising approach for training decentralized data located on local client devices while improving efficiency and privacy. However, the distribution and quantity of the training data on the clients' side may lead to significant challenges such as class imbalance and non-IID (non-independent and identically distributed) data, which could greatly impact the performance of the common model. While much effort has been devoted to helping FL models converge when encountering non-IID data, the imbalance issue has not been sufficiently addressed. In particular, as FL training is executed by exchanging gradients in an encrypted form, the training data is not completely observable to either clients or servers, and previous methods for class imbalance do not perform well for FL. Therefore, it is crucial to design new methods for detecting class imbalance in FL and mitigating its impact. In this work, we propose a monitoring scheme that can infer the composition of training data for each FL round, and design a new loss function -- \textbf{Ratio Loss} to mitigate the impact of the imbalance. Our experiments demonstrate the importance of acknowledging class imbalance and taking measures as early as possible in FL training, and the effectiveness of our method in mitigating the impact. Our method is shown to significantly outperform previous methods, while maintaining client privacy.
School of Management Engineering, Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院管理工程学院）
In clinical research, the lack of events of interest often necessitates imbalanced learning. One approach to resolve this obstacle is data integration or sharing, but due to privacy concerns neither is practical. Therefore, there is an increasing demand for a platform on which an analysis can be performed in a federated environment while maintaining privacy. However, it is quite challenging to develop a federated learning algorithm that can address both privacy-preserving and class imbalanced issues. In this study, we introduce a federated generative model learning platform for generating samp…
Federated learning is a distributed machine learning paradigm that trains a global model for prediction based on a number of local models at clients while local data privacy is preserved. Class imbalance is believed to be one of the factors that degrades the global model performance. However, there has been very little research on if and how class imbalance can affect the global performance. class imbalance in federated learning is much more complex than that in traditional non-distributed machine learning, due to different class imbalance situations at local clients. Class imbalance needs to …
Northeastern University（东北大学）；The University of Melbourne（墨尔本大学）；RMIT University（皇家墨尔本理工大学）
Federated learning, which allows multiple client devices in a network to jointly train a machine learning model without direct exposure of clients' data, is an emerging distributed learning technique due to its nature of privacy preservation. However, it has been found that models trained with federated learning usually have worse performance than their counterparts trained in the standard centralized learning mode, especially when the training data is imbalanced. In the context of federated learning, data imbalance may occur either locally one one client device, or globally across many device…
Framework for Co-distillation Driven Federated Learning to Address Class Imbalance in Healthcare
Indian Institute of Technology, Bombay（印度理工学院孟买分校）；Sardar Patel Institute of Technology（萨达尔·帕特尔理工学院）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
FedReLa: Imbalanced Federated Learning via Re-Labeling
School of Computing and Information Systems, University of Melbourne, Victoria, Australia（墨尔本大学计算机与信息系统学院）；School of Statistics and Data Science, LPMC, KLMDASR, and LEBPS, Nankai University（南开大学统计与数据科学学院）；School of Mathematics and Statistics, University of Melbourne, Victoria, Australia（墨尔本大学数学与统计学学院）
Fast Convergence through Distributed Augmentation for Class-Imbalanced Federated Learning
Amar Nath and Shashi Khosla School of Information Technology, Indian Institute of Technology Delhi（印度理工学院德里分校 Amar Nath and Shashi Khosla 信息技术学院）；Department of Electrical Engineering, Indian Institute of Technology Delhi（印度理工学院德里分校电气工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Chongqing University（重庆大学）；Beijing Institute of Control Engineering（北京控制工程研究所）
Federated learning (FL) is a distributed deep learning method which enables multiple participants, such as mobile phones and IoT devices, to contribute a neural network model while their private training data remains in local devices. This distributed approach is promising in the edge computing system where have a large corpus of decentralized data and require high privacy. However, unlike the common training dataset, the data distribution of the edge computing system is imbalanced which will introduce biases in the model training and cause a decrease in accuracy of federated learning applicat…
Department of Applied Mathematics and Theoretical Physics, University of Cambridge（应用数学与理论物理系，剑桥大学）；Department of Radiology, University of Cambridge（放射学系，剑桥大学）
The Federated Learning setting has a central server coordinating the training of a model on a network of devices. One of the challenges is variable training performance when the dataset has a class imbalance. In this paper, we address this by introducing a new loss function called Fed-Focal Loss. We propose to address the class imbalance by reshaping cross-entropy loss such that it down-weights the loss assigned to well-classified examples along the lines of focal loss. Additionally, by leveraging a tunable sampling framework, we take into account selective client model contributions on the ce…
FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity
Huazhong University of Science and Technology（华中科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Hong Kong University of Science and Technology（香港科技大学）
ShanghaiTech University（上海科技大学）；Shanghai Advanced Research Institute, Chinese Academy of Sciences (CAS)（中国科学院上海高等研究院）
Federated learning (FL) is a promising technique that enables a large amount of edge computing devices to collaboratively train a global learning model. Due to privacy concerns, the raw data on devices could not be available for centralized server. Constrained by the spectrum limitation and computation capacity, only a subset of devices can be engaged to train and transmit the trained model to centralized server for aggregation. Since the local data distribution varies among all devices, class imbalance problem arises along with the unfavorable client selection, resulting in a slow converge ra…
Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data
Washington State University（华盛顿州立大学）；Princeton University（普林斯顿大学）；IBM Research（IBM研究院）；University of Florida（佛罗里达大学）；University of California San Diego（加州大学圣迭戈分校）
Peking University（北京大学）；Key Laboratory of Machine Perception (Ministry of Education), Peking University（北京大学机器感知重点实验室（教育部））；National Institute of Health Data Science, Peking University（北京大学健康数据科学研究院）；Institute of Medical Technology, Health Science Center of Peking University（北京大学医学部医学技术研究院）
Since data is presented long-tailed in reality, it is challenging for Federated Learning (FL) to train across decentralized clients as practical applications. We present Global-Regularized Personalization (GRP-FED) to tackle the data imbalanced issue by considering a single global model and multiple local models for each client. With adaptive aggregation, the global model treats multiple clients fairly and mitigates the global long-tailed issue. Each local model is learned from the local data and aligns with its distribution for customization. To prevent the local model from just overfitting, …
With the ever increasing data deluge and the success of deep neural networks, the research of distributed deep learning has become pronounced. Two common approaches to achieve this distributed learning is synchronous and asynchronous weight update. In this manuscript, we have explored very simplistic synchronous weight update mechanisms. It has been seen that with an increasing number of worker nodes, the performance degrades drastically. This effect has been studied in the context of extreme imbalanced classification (e.g. outlier detection). In practical cases, the assumed conditions of i.i.…
Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction
Duke University（杜克大学）；University of Maryland, College Park（马里兰大学帕克分校）；George Mason University（乔治梅森大学）；Yale University（耶鲁大学）；The State University of New York at Buffalo（纽约州立大学布法罗分校）
Qatar University（卡塔尔大学）；Qualcomm Inc（高通公司）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
The usage of unmanned aerial vehicles (UAVs) in civil and military applications continues to increase due to the numerous advantages that they provide over conventional approaches. Despite the abundance of such advantages, it is imperative to investigate the performance of UAV utilization while considering their design limitations. This paper investigates the deployment of UAV swarms when each UAV carries a machine learning classification task. To avoid data exchange with ground-based processing nodes, a federated learning approach is adopted between a UAV leader and the swarm members to impro…
Mitigating Participation Imbalance Bias in Asynchronous Federated Learning
University of California, Riverside（加州大学河滨分校）；Indian Institute of Science（印度科学研究院）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；DEVCOM Army Research Laboratory（陆军研究实验室）
Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Republic of Korea（人工智能跨学科项目，首尔国立大学，韩国）；Department of Nuclear Medicine, Seoul National University College of Medicine, Republic of Korea（核医学系，首尔国立大学医学院，韩国）；Brightonix Imaging Inc. Seoul, Republic of Korea（Brightonix影像公司，首尔，韩国）
Microsoft Research（微软研究院）；University of Southern California（南加州大学）
We study a new form of federated learning where the clients train personalized local models and make predictions jointly with the server-side shared model. Using this new federated learning framework, the complexity of the central shared model can be minimized while still gaining all the performance benefits that joint training provides. Our framework is robust to data heterogeneity, addressing the slow convergence problem traditional federated learning methods face when the data is non-i.i.d. across clients. We test the theory empirically and find substantial performance gains over baselines.
Federated learning enables many local devices to train a deep learning model jointly without sharing the local data. Currently, most of federated training schemes learns a global model by averaging the parameters of local models. However, most of these training schemes suffer from high communication cost resulted from transmitting full local model parameters. Moreover, directly averaging model parameters leads to a significant performance degradation, due to the class-imbalanced non-iid data on different devices. Especially for the real life federated learning tasks involving extreme classific…
Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（教育部新一代人工智能技术及其交叉应用重点实验室（东南大学））
Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift
The Pennsylvania State University（宾夕法尼亚州立大学）；State University of New York at Albany（纽约州立大学阿尔巴尼分校）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；Dalian University of Technology（大连理工大学）
Federated learning (FL) aided health diagnostic models can incorporate data from a large number of personal edge devices (e.g., mobile phones) while keeping the data local to the originating devices, largely ensuring privacy. However, such a cross-device FL approach for health diagnostics still imposes many challenges due to both local data imbalance (as extreme as local data consists of a single disease class) and global data imbalance (the disease prevalence is generally low in a population). Since the federated server has no access to data distribution information, it is not trivial to solv…
Robust Federated Learning with Confidence-Weighted Filtering and GAN-Based Completion under Noisy and Incomplete Data
Computer Engineering Department, İstanbul Commerce University（伊斯坦布尔商业大学计算机工程系）；Turkcell Teknoloji（Turkcell技术公司）；Grid Communications and Security Group, Electrification and Energy Infrastructures Division, Oak Ridge National Laboratory（橡树岭国家实验室电网安全与能源基础设施部门）
针对联邦学习中的噪声标签、类别缺失与不均衡问题，提出结合自适应噪声清洗、条件 GAN 合成补全和鲁棒训练的隐私保护框架，并在 MNIST 与 Fashion-MNIST 上验证其性能提升。
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Duke University（杜克大学）
Federated learning (FL) allows multiple clients to collaboratively train a deep learning model. One major challenge of FL is when data distribution is heterogeneous, i.e., differs from one client to another. Existing personalized FL algorithms are only applicable to narrow cases, e.g., one or two data classes per client, and therefore they do not satisfactorily address FL under varying levels of data heterogeneity. In this paper, we propose a novel framework, called DisTrans, to improve FL performance (i.e., model accuracy) via train and test-time distributional transformations along with a do…
Federated learning (FL) has gain growing interests for its capability of learning from distributed data sources collectively without the need of accessing the raw data samples across different sources. So far FL research has mostly focused on improving the performance, how the algorithmic disparity will be impacted for the model learned from FL and the impact of algorithmic disparity on the utility inconsistency are largely unexplored. In this paper, we propose an FL framework to jointly consider performance consistency and algorithmic fairness across different local clients (data sources). We…
We present a federated learning approach for learning a client adaptable, robust model when data is non-identically and non-independently distributed (non-IID) across clients. By simulating heterogeneous clients, we show that adding learned client-specific conditioning improves model performance, and the approach is shown to work on balanced and imbalanced data set from both audio and image domains. The client adaptation is implemented by a conditional gated activation unit and is particularly beneficial when there are large differences between the data distribution for each client, a common s…
Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators
The Hong Kong University of Science and Technology（香港科技大学）；SenseTime Research（商汤科技研究院）；Sony AI（索尼人工智能）；The Hong Kong Polytechnic University（香港理工大学）；Northwestern Polytechnical University（西北工业大学）
KAIST AI（韩国科学技术院人工智能研究院）；NAVER AI LAB（NAVER人工智能实验室）
Although federated learning has made awe-inspiring advances, most studies have assumed that the client's data are fully labeled. However, in a real-world scenario, every client may have a significant amount of unlabeled instances. Among the various approaches to utilizing unlabeled data, a federated active learning framework has emerged as a promising solution. In the decentralized setting, there are two types of available query selector models, namely 'global' and 'local-only' models, but little literature discusses their performance dominance and its causes. In this work, we first demonstrat…
In this paper we envision a federated learning (FL) scenario in service of amending the performance of autonomous road vehicles, through a drone traffic monitor (DTM), that also acts as an orchestrator. Expecting non-IID data distribution, we focus on the issue of accelerating the learning of a particular class of critical object (CO), that may harm the nominal operation of an autonomous vehicle. This can be done through proper allocation of the wireless resources for addressing learner and data heterogeneity. Thus, we propose a reactive method for the allocation of wireless resources, that ha…
Peking University（北京大学）；Renmin University of China（中国人民大学）；Shanghai AI Lab（上海人工智能实验室）；Tencent Inc.（腾讯公司）；Tsinghua University（清华大学）
Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy. However, federated learning faces severe optimization difficulty when training samples are not independently and identically distributed (non-i.i.d.). In this paper, we point out that the client sampling practice plays a decisive role in the aforementioned optimization difficulty. We find that the negative client sampling will cause the merged data distribution of currently sampled clients heavily inconsistent with that…
FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarios
Shenzhen University（深圳大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Federated learning aims to collaboratively train models without accessing their client's local private data. The data may be Non-IID for different clients and thus resulting in poor performance. Recently, personalized federated learning (PFL) has achieved great success in handling Non-IID data by enforcing regularization in local optimization or improving the model aggregation scheme on the server. However, most of the PFL approaches do not take into account the unfair competition issue caused by the imbalanced data distribution and lack of positive samples for some classes in each client. To …
INO-SGD: Addressing Utility Imbalance under Individualized Differential Privacy
Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）；Agency for Science, Technology and Research (A*STAR), Singapore（新加坡科技研究局）
