We study a family of algorithms, which we refer to as local update methods, generalizing many federated and meta-learning algorithms. We prove that for quadratic models, local update methods are equivalent to first-order optimization on a surrogate loss we exactly characterize. Moreover, fundamental algorithmic choices (such as learning rates) explicitly govern a trade-off between the condition number of the surrogate loss and its alignment with the true loss. We derive novel convergence rates showcasing these trade-offs and highlight their importance in communication-limited settings. Using these insights, we are able to compare local update methods based on their convergence/accuracy trade-off, not just their convergence to critical points of the empirical loss. Our results shed new light on a broad range of phenomena, including the efficacy of server momentum in federated learning and the impact of proximal client updates.
Spiking Neural Networks (SNNs) are biologically inspired alternatives to conventional Artificial Neural Networks (ANNs). Despite promising preliminary results, the trade-offs in the training of SNNs in a distributed scheme are not well understood. Here, we consider SNNs in a federated learning setting where a high-quality global model is created by aggregating multiple local models from the clients without sharing any data. We investigate federated learning for training multiple SNNs at clients when two mechanisms reduce the uplink communication cost: i) random masking of the model updates sen…
Beijing University of Posts and Telecommunications（北京邮电大学）
The proliferation of the Internet of Things (IoT) and widespread use of devices with sensing, computing, and communication capabilities have motivated intelligent applications empowered by artificial intelligence. The classical artificial intelligence algorithms require centralized data collection and processing which are challenging in realistic intelligent IoT applications due to growing data privacy concerns and distributed datasets. Federated Learning (FL) has emerged as a distributed privacy-preserving learning framework that enables IoT devices to train global model through sharing model…
Queen’s University（女王大学）；Pukyong National University（釜庆国立大学）
Federated Learning (FL) has recently received a lot of attention for large-scale privacy-preserving machine learning. However, high communication overheads due to frequent gradient transmissions decelerate FL. To mitigate the communication overheads, two main techniques have been studied: (i) local update of weights characterizing the trade-off between communication and computation and (ii) gradient compression characterizing the trade-off between communication and precision. To the best of our knowledge, studying and balancing those two trade-offs jointly and dynamically while considering the…
Federated learning (FL) allows to train a massive amount of data privately due to its decentralized structure. Stochastic gradient descent (SGD) is commonly used for FL due to its good empirical performance, but sensitive user information can still be inferred from weight updates shared during FL iterations. We consider Gaussian mechanisms to preserve local differential privacy (LDP) of user data in the FL model with SGD. The trade-offs between user privacy, global utility, and transmission rate are proved by defining appropriate metrics for FL with LDP. Compared to existing results, the query…
Telefonica Research（西班牙电信研究院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Federated learning (FL), where data remains at the federated clients, and where only gradient updates are shared with a central aggregator, was assumed to be private. Recent work demonstrates that adversaries with gradient-level access can mount successful inference and reconstruction attacks. In such settings, differentially private (DP) learning is known to provide resilience. However, approaches used in the status quo (\ie central and local DP) introduce disparate utility vs. privacy trade-offs. In this work, we take the first step towards mitigating such trade-offs through {\em hierarchica…
Beyond Trade-offs: A Unified Framework for Privacy, Robustness, and Communication Efficiency in Federated Learning
School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）；Electrical Engineering and Computer Science Department, Technische Universität Berlin（电气工程与计算机科学系，柏林技术大学）
Federated Learning in Genetics: Extended Analysis of Accuracy, Performance and Privacy Trade-offs
Dept. of Computer Science（计算机科学系）；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（可扩展数据分析与人工智能中心（ScaDS.AI）德累斯顿/莱比锡）；Leipzig University（莱比锡大学）
Learning Heterogeneous Performance-Fairness Trade-offs in Federated Learning
Department of Computer Science and Engineering and the Research Institute of Trustworthy Autonomous Systems at Southern University of Science and Technology（计算机科学与工程系和可信自主系统研究 institute at 南方科技大学）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Google Research（谷歌研究院）
We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework. Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each roun…
Privacy, Robustness, and Fairness Trade-offs in Federated Intrusion Detection: Geometric Indistinguishability at the Aggregation Interface
Bangladesh University of Business and Technology (BUBT)（孟加拉国商业与技术大学（BUBT））；Jagannath University（贾格纳特大学）；University of New South Wales (UNSW)（新南威尔士大学（UNSW））
Google Research（谷歌研究院）；Carnegie Mellon University（卡内基梅隆大学）
We introduce the multi-dimensional Skellam mechanism, a discrete differential privacy mechanism based on the difference of two independent Poisson random variables. To quantify its privacy guarantees, we analyze the privacy loss distribution via a numerical evaluation and provide a sharp bound on the Rényi divergence between two shifted Skellam distributions. While useful in both centralized and distributed privacy applications, we investigate how it can be applied in the context of federated learning with secure aggregation under communication constraints. Our theoretical findings and extensi…
University of Maryland（马里兰大学）；IBM Research（IBM研究院）；University of Virginia（弗吉尼亚大学）
Due to its distributed methodology alongside its privacy-preserving features, Federated Learning (FL) is vulnerable to training time adversarial attacks. In this study, our focus is on backdoor attacks in which the adversary's goal is to cause targeted misclassifications for inputs embedded with an adversarial trigger while maintaining an acceptable performance on the main learning task at hand. Contemporary defenses against backdoor attacks in federated learning require direct access to each individual client's update which is not feasible in recent FL settings where Secure Aggregation is dep…
Massachusetts Institute of Technology（麻省理工学院）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
In Federated Learning, we aim to train models across multiple computing units (users), while users can only communicate with a common central server, without exchanging their data samples. This mechanism exploits the computational power of all users and allows users to obtain a richer model as their models are trained over a larger set of data points. However, this scheme only develops a common output for all the users, and, therefore, it does not adapt the model to each user. This is an important missing feature, especially given the heterogeneity of the underlying data distribution for vario…
Federated learning (FL) aims to train machine learning models in the decentralized system consisting of an enormous amount of smart edge devices. Federated averaging (FedAvg), the fundamental algorithm in FL settings, proposes on-device training and model aggregation to avoid the potential heavy communication costs and privacy concerns brought by transmitting raw data. However, through theoretical analysis we argue that 1) the multiple steps of local updating will result in gradient biases and 2) there is an inconsistency between the expected target distribution and the optimization objectives…
Performance and Energy Trade-Off Analysis of Hierarchical Federated Learning for Plant Disease Classification
University of Zagreb（扎格雷布大学）；University of Thessaly（塞萨洛尼基大学）；University of Glasgow（格拉斯哥大学）；Trinity College Dublin（都柏林信任学院）；Italian National Research Council（意大利国家研究委员会）
University of Washington（华盛顿大学）；Google Research（谷歌研究院）
Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number o…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Duke University（杜克大学）
Federated learning (FL) allows multiple clients to collaboratively train a deep learning model. One major challenge of FL is when data distribution is heterogeneous, i.e., differs from one client to another. Existing personalized FL algorithms are only applicable to narrow cases, e.g., one or two data classes per client, and therefore they do not satisfactorily address FL under varying levels of data heterogeneity. In this paper, we propose a novel framework, called DisTrans, to improve FL performance (i.e., model accuracy) via train and test-time distributional transformations along with a do…
To enable large-scale machine learning in bandwidth-hungry environments such as wireless networks, significant progress has been made recently in designing communication-efficient federated learning algorithms with the aid of communication compression. On the other end, privacy-preserving, especially at the client level, is another important desideratum that has not been addressed simultaneously in the presence of advanced communication compression techniques yet. In this paper, we propose a unified framework that enhances the communication efficiency of private federated learning with communi…
The performance of federated learning systems is bottlenecked by communication costs and training variance. The communication overhead problem is usually addressed by three communication-reduction techniques, namely, model compression, partial device participation, and periodic aggregation, at the cost of increased training variance. Different from traditional distributed learning systems, federated learning suffers from data heterogeneity (since the devices sample their data from possibly different distributions), which induces additional variance among devices during training. Various varian…
We describe the design of our federated task processing system. Originally, the system was created to support two specific federated tasks: evaluation and tuning of on-device ML systems, primarily for the purpose of personalizing these systems. In recent years, support for an additional federated task has been added: federated learning (FL) of deep neural networks. To our knowledge, only one other system has been described in literature that supports FL at scale. We include comparisons to that system to help discuss design decisions and attached trade-offs. Finally, we describe two specific la…
The Chinese University of Hong Kong（香港中文大学）；Huawei Technologies Co. Ltd（华为技术有限公司）
The high cost of communicating gradients is a major bottleneck for federated learning, as the bandwidth of the participating user devices is limited. Existing gradient compression algorithms are mainly designed for data centers with high-speed network and achieve $O(\sqrt{d} \log d)$ per-iteration communication cost at best, where $d$ is the size of the model. We propose hyper-sphere quantization (HSQ), a general framework that can be configured to achieve a continuum of trade-offs between communication efficiency and gradient accuracy. In particular, at the high compression ratio end, HSQ pro…
Center for Frontier AI Research（前沿人工智能研究中心）；A*STAR（新加坡科技研究局）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）
Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model. However, this would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous. As a result, current federated learning methods often require a large number of communication rounds in training. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning. In this work, we first introduce a meta knowledge representati…
Green Federated Learning via Carbon-Aware Client and Time Slot Scheduling
Inria Centre at Université Côte d’Azur（法国国家信息与自动化技术研究院（Inria）亚维农分校）；Accenture（埃森哲）；Department of Electrical Engineering (ISY), Linköping University（利厄普大学电气工程系）
University of Southern California（南加州大学）；University of California, Riverside（加州大学河滨分校）
Secure federated learning is a privacy-preserving framework to improve machine learning models by training over large volumes of data collected by mobile users. This is achieved through an iterative process where, at each iteration, users update a global model using their local datasets. Each user then masks its local model via random keys, and the masked models are aggregated at a central server to compute the global model for the next iteration. As the local models are protected by random masks, the server cannot observe their true values. This presents a major challenge for the resilience o…
Given the abundance and ease of access of personal data today, individual privacy has become of paramount importance, particularly in the healthcare domain. In this work, we aim to utilise patient data extracted from multiple hospital data centres to train a machine learning model without sacrificing patient privacy. We develop a scheduling algorithm in conjunction with a student-teacher algorithm that is deployed in a federated manner. This allows a central model to learn from batches of data at each federal node. The teacher acts between data centres to update the main task (student) algorit…
LinkedIn Corporation（领英公司）；University of Michigan（密歇根大学）；RightScope Inc.（RightScope公司）
Cross-device federated learning (FL) has been well-studied from algorithmic, system scalability, and training speed perspectives. Nonetheless, moving from centralized training to cross-device FL for millions or billions of devices presents many risks, including performance loss, developer inertia, poor user experience, and unexpected application failures. In addition, the corresponding infrastructure, development costs, and return on investment are difficult to estimate. In this paper, we present a device-cloud collaborative FL platform that integrates with an existing machine learning platfor…
Distributed deep learning frameworks like federated learning (FL) and its variants are enabling personalized experiences across a wide range of web clients and mobile/IoT devices. However, FL-based frameworks are constrained by computational resources at clients due to the exploding growth of model parameters (eg. billion parameter model). Split learning (SL), a recent framework, reduces client compute load by splitting the model training between client and server. This flexibility is extremely useful for low-compute setups but is often achieved at cost of increase in bandwidth consumption and…
In response to growing concerns about user privacy, federated learning has emerged as a promising tool to train statistical models over networks of devices while keeping data localized. Federated learning methods run training tasks directly on user devices and do not share the raw user data with third parties. However, current methods still share model updates, which may contain private information (e.g., one's weight and height), during the training process. Existing efforts that aim to improve the privacy of federated learning make compromises in one or more of the following key areas: perfo…
MRI-based and metabolomics-based age scores act synergetically for mortality prediction shown by multi-cohort federated learning
Maastricht University Medical Centre +（马斯特里赫特大学医学中心）；Leiden University Medical Center（莱顿大学医学中心）；Delft University of Technology（代尔夫特理工大学）；Erasmus MC - University Medical Center Rotterdam（伊拉斯姆斯大学医学中心鹿特丹分校）；Leiden University（莱顿大学）
Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through ex…
Blockchain-empowered Federated Learning for Healthcare Metaverses: User-centric Incentive Mechanism with Optimal Data Freshness
Guangdong University of Technology（广东工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）
Federated Learning (FL), as a rapidly evolving privacy-preserving collaborative machine learning paradigm, is a promising approach to enable edge intelligence in the emerging Industrial Metaverse. Even though many successful use cases have proved the feasibility of FL in theory, in the industrial practice of Metaverse, the problems of non-independent and identically distributed (non-i.i.d.) data, learning forgetting caused by streaming industrial data, and scarce communication bandwidth remain key barriers to realize practical FL. Facing the above three challenges simultaneously, this paper pr…
Personalized Federated Learning via Amortized Bayesian Meta-Learning
University of Electronic Science and Technology of China（电子科技大学）；Nanjing Audit University（南京审计大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng National Lab（鹏城国家实验室）
Georgia Institute of Technology（佐治亚理工学院）；IBM Research Almaden（IBM阿尔马登研究中心）
Federated learning facilitates the collaborative training of models without the sharing of raw data. However, recent attacks demonstrate that simply maintaining data locality during training processes does not provide sufficient privacy guarantees. Rather, we need a federated learning system capable of preventing inference over both the messages exchanged during training and the final trained model while ensuring the resulting model also has acceptable predictive accuracy. Existing federated learning approaches either use secure multiparty computation (SMC) which is vulnerable to inference or …
