Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating client. However, in real applications, the devices of clients are usually heterogeneous, and have different computing power. Although big models like BERT have achieved huge success in AI, it is difficult to apply them to heterogeneous FL with weak clients. The straightforward solutions like removing the weak clients or using a small model to fit all clients will lead to some problems, such as under-representation of dropped clients and inferior accuracy due to data loss or limited model representation ability. In this work, we propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities, bigger models for powerful clients and smaller ones for weak clients. We also propose an effective method to share the knowledge among multiple local models with different sizes. In this way, all the clients can participate in the model learning in FL, and the final model can be big and powerful enough. Besides, we propose a momentum knowledge distillation method to better transfer knowledge in big models on powerful clients to the small models on weak clients. Extensive experiments on many real-world benchmark datasets demonstrate the effectiveness of the proposed method in learning accurate models from clients with heterogeneous devices under the FL framework.
Federated learning (FL) is a distributed learning technique that trains a shared model over distributed data in a privacy-preserving manner. Unfortunately, FL's performance degrades when there is (i) variability in client characteristics in terms of computational and memory resources (system heterogeneity) and (ii) non-IID data distribution across clients (statistical heterogeneity). For example, slow clients get dropped in FL schemes, such as Federated Averaging (FedAvg), which not only limits overall learning but also biases results towards fast clients. We propose FedPrune; a system that ta…
Inclusive Federated Learning Through Compliance-Weighted Noise Allocation in Healthcare AI
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；Bilkent University（比尔肯特大学）；Indiana University（印第安纳大学）；Heidelberg University Hospital（海德堡大学医院）；MLCommons
University of Washington（华盛顿大学）；Kennesaw State University（肯尼索州立大学）
Federated learning involves training statistical models over remote devices such as mobile phones while keeping data localized. Training in heterogeneous and potentially massive networks introduces opportunities for privacy-preserving data analysis and diversifying these models to become more inclusive of the population. Federated learning can be viewed as a unique opportunity to bring fairness and parity to many existing models by enabling model training to happen on a diverse set of participants and on data that is generated regularly and dynamically. In this paper, we discuss the current me…
George Mason University（乔治梅森大学）；Stevens Institute of Technology（斯蒂文斯理工学院）
Federated learning (FL) is a machine learning paradigm where a shared central model is learned across distributed edge devices while the training data remains on these devices. Federated Averaging (FedAvg) is the leading optimization method for training non-convex models in this setting with a synchronized protocol. However, the assumptions made by FedAvg are not realistic given the heterogeneity of devices. In particular, the volume and distribution of collected data vary in the training process due to different sampling rates of edge devices. The edge devices themselves also vary in their av…
Hong Kong Baptist University（香港浸会大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Tsinghua University（清华大学）
Privacy and security concerns in real-world applications have led to the development of adversarially robust federated models. However, the straightforward combination between adversarial training and federated learning in one framework can lead to the undesired robustness deterioration. We discover that the attribution behind this phenomenon is that the generated adversarial data could exacerbate the data heterogeneity among local clients, making the wrapped federated learning perform poorly. To deal with this problem, we propose a novel framework called Slack Federated Adversarial Training (…
Lebanese American University（黎巴嫩美国大学）；Zayed University（扎耶德大学）；New York University（纽约大学）
Autism Spectrum Disorder (ASD) is a neuro-developmental syndrome resulting from alterations in the embryological brain before birth. This disorder distinguishes its patients by special socially restricted and repetitive behavior in addition to specific behavioral traits. Hence, this would possibly deteriorate their social behavior among other individuals, as well as their overall interaction within their community. Moreover, medical research has proved that ASD also affects the facial characteristics of its patients, making the syndrome recognizable from distinctive signs within an individual'…
Federated Learning (FL) is a decentralized machine learning protocol that allows a set of participating agents to collaboratively train a model without sharing their data. This makes FL particularly suitable for settings where data privacy is desired. However, it has been observed that the performance of FL is closely tied with the local data distributions of agents. Particularly, in settings where local data distributions vastly differ among agents, FL performs rather poorly with respect to the centralized training. To address this problem, we hypothesize the reasons behind the performance de…
Michigan State University（密歇根州立大学）；The Ohio State University（俄亥俄州立大学）；Google Research（谷歌研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling su…
Distributed algorithms have been playing an increasingly important role in many applications such as machine learning, signal processing, and control. Significant research efforts have been devoted to developing and analyzing new algorithms for various applications. In this work, we provide a fresh perspective to understand, analyze, and design distributed optimization algorithms. Through the lens of multi-rate feedback control, we show that a wide class of distributed algorithms, including popular decentralized/federated schemes, can be viewed as discretizing a certain continuous-time feedbac…
Training deep learning models on in-home IoT sensory data is commonly used to recognise human activities. Recently, federated learning systems that use edge devices as clients to support local human activity recognition have emerged as a new paradigm to combine local (individual-level) and global (group-level) models. This approach provides better scalability and generalisability and also offers better privacy compared with the traditional centralised analysis and learning models. The assumption behind federated learning, however, relies on supervised learning on clients. This requires a large…
In this paper, we question the rationale behind propagating large numbers of parameters through a distributed system during federated learning. We start by examining the rank characteristics of the subspace spanned by gradients across epochs (i.e., the gradient-space) in centralized model training, and observe that this gradient-space often consists of a few leading principal components accounting for an overwhelming majority (95-99%) of the explained variance. Motivated by this, we propose the "Look-back Gradient Multiplier" (LBGM) algorithm, which exploits this low-rank property to enable gr…
WeBank Co. Ltd.（微众银行）；Hong Kong University of Science and Technology（香港科技大学）；Extreme Vision Co. Ltd.（极视角科技有限公司）；South China University of Technology（华南理工大学）；Shenzhen University（深圳大学）
Federated learning is a new machine learning paradigm which allows data parties to build machine learning models collaboratively while keeping their data secure and private. While research efforts on federated learning have been growing tremendously in the past two years, most existing works still depend on pre-existing public datasets and artificial partitions to simulate data federations due to the lack of high-quality labeled data generated from real-world edge applications. Consequently, advances on benchmark and model evaluations for federated learning have been lagging behind. In this pa…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Federated learning has been spotlighted as a way to train neural networks using distributed data with no need for individual nodes to share data. Unfortunately, it has also been shown that adversaries may be able to extract local data contents off model parameters transmitted during federated learning. A recent solution based on the secure aggregation primitive enabled privacy-preserving federated learning, but at the expense of significant extra communication/computational resources. In this paper, we propose a low-complexity scheme that provides data privacy using substantially reduced commu…
Singapore University of Technology and Design（新加坡科技设计大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sichuan University（四川大学）；The Hong Kong Polytechnic University（香港理工大学）
As a novel distributed learning paradigm, federated learning (FL) faces serious challenges in dealing with massive clients with heterogeneous data distribution and computation and communication resources. Various client-variance-reduction schemes and client sampling strategies have been respectively introduced to improve the robustness of FL. Among others, primal-dual algorithms such as the alternating direction of method multipliers (ADMM) have been found being resilient to data distribution and outperform most of the primal-only FL algorithms. However, the reason behind remains a mystery sti…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of Moratuwa（莫拉图瓦大学）
We propose how a developing country like Sri Lanka can benefit from privacy-enabled machine learning techniques such as Federated Learning to detect road conditions using crowd-sourced data collection and proposed the idea of implementing a Digital Twin for the national road system in Sri Lanka. Developing countries such as Sri Lanka are far behind in implementing smart road systems and smart cities compared to the developed countries. The proposed work discussed in this paper matches the UN Sustainable Development Goal (SDG) 9: "Build Resilient Infrastructure, Promote Inclusive and Sustainabl…
Data heterogeneity is an inherent challenge that hinders the performance of federated learning (FL). Recent studies have identified the biased classifiers of local models as the key bottleneck. Previous attempts have used classifier calibration after FL training, but this approach falls short in improving the poor feature representations caused by training-time classifier biases. Resolving the classifier bias dilemma in FL requires a full understanding of the mechanisms behind the classifier. Recent advances in neural collapse have shown that the classifiers and feature prototypes under perfec…
Quantum Federated Learning: A Comprehensive Survey
Department of Electrical and Computer Engineering, The University of Alabama in Huntsville, USA（电气与计算机工程系，美国阿拉巴马大学亨茨维尔分校）；Faculty of Engineering and Applied Science, Memorial University, Canada（工程与应用科学学院，加拿大纪念大学）；College of Computer Science and Engineering, Nanyang Technological University, Singapore（计算机科学与工程学院，新加坡南洋理工大学）
Recent studies have shown that federated learning (FL) is vulnerable to poisoning attacks that inject a backdoor into the global model. These attacks are effective even when performed by a single client, and undetectable by most existing defensive techniques. In this paper, we propose Backdoor detection via Feedback-based Federated Learning (BAFFLE), a novel defense to secure FL against backdoor attacks. The core idea behind BAFFLE is to leverage data of multiple clients not only for training but also for uncovering model poisoning. We exploit the availability of diverse datasets at the variou…
University of Southern California（南加州大学）；Hong Kong Baptist University（香港浸会大学）；Tencent AI Lab（腾讯AI Lab）
Federated Learning (FL) is a distributed learning paradigm that can learn a global or personalized model from decentralized datasets on edge devices. However, in the computer vision domain, model performance in FL is far behind centralized training due to the lack of exploration in diverse tasks with a unified FL framework. FL has rarely been demonstrated effectively in advanced computer vision tasks such as object detection and image segmentation. To bridge the gap and facilitate the development of FL for computer vision tasks, in this work, we propose a federated learning library and benchma…
FLEDGE: Ledger-based Federated Learning Resilient to Inference and Backdoor Attacks
The University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德谷分校）；Technical University of Darmstadt（达姆施塔特工业大学）；KOBIL GmbH（KOBIL有限公司）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Key Lab of High-Confidence Software Technology, MoE (Peking University)（高可信软件技术教育部重点实验室（北京大学））；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Federated learning (FL) is an emerging promising privacy-preserving machine learning paradigm and has raised more and more attention from researchers and developers. FL keeps users' private data on devices and exchanges the gradients of local models to cooperatively train a shared Deep Learning (DL) model on central custodians. However, the security and fault tolerance of FL have been increasingly discussed, because its central custodian mechanism or star-shaped architecture can be vulnerable to malicious attacks or software failures. To address these problems, Swarm Learning (SL) introduces a…
Privacy-Preserving Personalized Federated Learning for Distributed Photovoltaic Disaggregation under Statistical Heterogeneity
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University（墨尔本大学数据科学与人工智能系）；Shenzhen Institute for Advanced Study of UESTC（电子科技大学深圳研究院）
Federated learning, where algorithms are trained across multiple decentralized devices without sharing local data, is increasingly popular in distributed machine learning practice. Typically, a graph structure $G$ exists behind local devices for communication. In this work, we consider parameter estimation in federated learning with data distribution and communication heterogeneity, as well as limited computational capacity of local devices. We encode the distribution heterogeneity by parametrizing distributions on local devices with a set of distinct $p$-dimensional vectors. We then propose t…
High-Energy Concentration for Federated Learning in Frequency Domain
State Key Laboratory of Integrated Services Networks, Xidian University（信息服务网络国家重点实验室，西安电子科技大学）；College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）
An Inclusive and Lightweight Approach to Federated Continual Learning for Cultural Heritage
Information Technologies Institute, Centre for Research and Technology Hellas（希腊研究与技术中心信息技术研究所）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）
Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology（香港科技大学跨学科研究学院）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Samsung Research America, inc.（三星美国研究公司）
Learning the similarity between structured data, especially the graphs, is one of the essential problems. Besides the approach like graph kernels, Gromov-Wasserstein (GW) distance recently draws big attention due to its flexibility to capture both topological and feature characteristics, as well as handling the permutation invariance. However, structured data are widely distributed for different data mining and machine learning applications. With privacy concerns, accessing the decentralized data is limited to either individual clients or different silos. To tackle these issues, we propose a p…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Pennsylvania（宾夕法尼亚大学）
The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at the server, is perhaps the most commonly used method in Federated Learning. Notwithstanding its simplicity, several empirical studies have illustrated that the output model of FedAvg, after a few fine-tuning steps, leads to a model that generalizes well to new unseen tasks. This surprising performance of such a simple method, however, is not fully understood from a theoretical point of view. In this paper, we formally…
National University of Singapore（新加坡国立大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Collaborative (federated) learning enables multiple parties to train a model without sharing their private data, but through repeated sharing of the parameters of their local models. Despite its advantages, this approach has many known privacy and security weaknesses and performance overhead, in addition to being limited only to models with homogeneous architectures. Shared parameters leak a significant amount of information about the local (and supposedly private) datasets. Besides, federated learning is severely vulnerable to poisoning attacks, where some participants can adversarially influ…
