In this paper, we quantitatively compare these two effective communication schemes, i.e., digital and analog ones, for wireless federated learning (FL) over resource-constrained networks, highlighting their essential differences as well as their respective application scenarios. We first examine both digital and analog transmission methods, together with a unified and fair comparison scheme under practical constraints. A universal convergence analysis under various imperfections is established for FL performance evaluation in wireless networks. These analytical results reveal that the fundamental difference between the two paradigms lies in whether communication and computation are jointly designed or not. The digital schemes decouple the communication design from specific FL tasks, making it difficult to support simultaneous uplink transmission of massive devices with limited bandwidth. In contrast, the analog communication allows over-the-air computation (AirComp), thus achieving efficient spectrum utilization. However, computation-oriented analog transmission reduces power efficiency, and its performance is sensitive to computational errors. Finally, numerical simulations are conducted to verify these theoretical observations.
In the past few decades, machine learning has revolutionized data processing for large scale applications. Simultaneously, increasing privacy threats in trending applications led to the redesign of classical data training models. In particular, classical machine learning involves centralized data training, where the data is gathered, and the entire training process executes at the central server. Despite significant convergence, this training involves several privacy threats on participants' data when shared with the central cloud server. To this end, federated learning has achieved significan…
In this paper, we consider federated learning (FL) over a noisy fading multiple access channel (MAC), where an edge server aggregates the local models transmitted by multiple end devices through over-the-air computation (AirComp). To realize efficient analog federated learning over wireless channels, we propose an AirComp-based FedSplit algorithm, where a threshold-based device selection scheme is adopted to achieve reliable local model uploading. In particular, we analyze the performance of the proposed algorithm and prove that the proposed algorithm linearly converges to the optimal solution…
Federated learning (FL) enables workers to learn a model collaboratively by using their local data, with the help of a parameter server (PS) for global model aggregation. The high communication cost for periodic model updates and the non-independent and identically distributed (i.i.d.) data become major bottlenecks for FL. In this work, we consider analog aggregation to scale down the communication cost with respect to the number of workers, and introduce data redundancy to the system to deal with non-i.i.d. data. We propose an online energy-aware dynamic worker scheduling policy, which maximi…
Wireless connectivity is instrumental in enabling scalable federated learning (FL), yet wireless channels bring challenges for model training, in which channel randomness perturbs each worker's model update while multiple workers' updates incur significant interference under limited bandwidth. To address these challenges, in this work we formulate a novel constrained optimization problem, and propose an FL framework harnessing wireless channel perturbations and interference for improving privacy, bandwidth-efficiency, and scalability. The resultant algorithm is coined analog federated ADMM (A-…
Spectral Co-Distillation for Personalized Federated Learning
Singapore University of Technology and Design (SUTD)（新加坡科技设计大学）；Zhejiang University/University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟校区联合学院，浙江大学）
Perfectly-Private Analog Secure Aggregation in Federated Learning
Department of Electrical Engineering, Chalmers University of Technology, Sweden（瑞典查尔姆斯理工大学电子工程系）；Department of Mathematics and Systems Analysis, Aalto University, Finland（芬兰阿尔托大学数学与系统分析系）
Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（皇家理工学院电气工程与计算机科学学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）
Princeton University（普林斯顿大学）；Imperial College London（伦敦帝国理工学院）
We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD). We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its …
The use of collaborative and decentralized machine learning techniques such as federated learning have the potential to enable the development and deployment of clinical risk predictions models in low-resource settings without requiring sensitive data be shared or stored in a central repository. This process necessitates communication of model weights or updates between collaborating entities, but it is unclear to what extent patient privacy is compromised as a result. To gain insight into this question, we study the efficacy of centralized versus federated learning in both private and non-pri…
Princeton University（普林斯顿大学）；Imperial College London（帝国理工学院）
We study federated learning (FL), where power-limited wireless devices utilize their local datasets to collaboratively train a global model with the help of a remote parameter server (PS). The PS has access to the global model and shares it with the devices for local training, and the devices return the result of their local updates to the PS to update the global model. This framework requires downlink transmission from the PS to the devices and uplink transmission from the devices to the PS. The goal of this study is to investigate the impact of the bandwidth-limited shared wireless medium in…
Federated learning (FL) is a distributed learning approach where a set of end-user devices participate in the learning process by acting on their isolated local data sets. Here, we process local data sets of users where worst-case optimization theory is used to reformulate the FL problem where the impact of local data sets in training phase is considered as an uncertain function bounded in a closed uncertainty region. This representation allows us to compare the performance of FL with its centralized counterpart, and to replace the uncertain function with a concept of protection functions lead…
Bilkent University（比尔肯特大学）；Huawei Turkey Research and Development Center (HTRDC)（华为土耳其研发中心）
We study collaborative machine learning systems where a massive dataset is distributed across independent workers which compute their local gradient estimates based on their own datasets. Workers send their estimates through a multipath fading multiple access channel with orthogonal frequency division multiplexing to mitigate the frequency selectivity of the channel. We assume that there is no channel state information (CSI) at the workers, and the parameter server (PS) employs multiple antennas to align the received signals. To reduce the power consumption and the hardware costs, we employ co…
Training ML models which are fair across different demographic groups is of critical importance due to the increased integration of ML in crucial decision-making scenarios such as healthcare and recruitment. Federated learning has been viewed as a promising solution for collaboratively training machine learning models among multiple parties while maintaining the privacy of their local data. However, federated learning also poses new challenges in mitigating the potential bias against certain populations (e.g., demographic groups), as this typically requires centralized access to the sensitive …
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We establish a connection between federated learning, a concept from machine learning, and mean-field games, a concept from game theory and control theory. In this analogy, the local federated learners are considered as the players and the aggregation of the gradients in a central server is the mean-field effect. We present federated learning as a differential game and discuss the properties of the equilibrium of this game. We hope this novel view to federated learning brings together researchers from these two distinct areas to work on fundamental problems of large-scale distributed and priva…
Machine learning for hybrid beamforming has been extensively studied by using centralized machine learning (CML) techniques, which require the training of a global model with a large dataset collected from the users. However, the transmission of the whole dataset between the users and the base station (BS) is computationally prohibitive due to limited communication bandwidth and privacy concerns. In this work, we introduce a federated learning (FL) based framework for hybrid beamforming, where the model training is performed at the BS by collecting only the gradients from the users. We design …
Andalusian Research Institute in Data Science and Computational Intelligence（安达卢西亚数据科学与计算智能研究院）；University of Granada（格拉纳达大学）
Federated learning is a machine learning paradigm that emerges as a solution to the privacy-preservation demands in artificial intelligence. As machine learning, federated learning is threatened by adversarial attacks against the integrity of the learning model and the privacy of data via a distributed approach to tackle local and global learning. This weak point is exacerbated by the inaccessibility of data in federated learning, which makes harder the protection against adversarial attacks and evidences the need to furtherance the research on defence methods to make federated learning a real…
Federated learning (FL) is an attractive paradigm for making use of rich distributed data while protecting data privacy. Nonetheless, nonideal communication links and limited transmission resources may hinder the implementation of fast and accurate FL. In this paper, we study joint optimization of communications and FL based on analog aggregation transmission in realistic wireless networks. We first derive closed-form expressions for the expected convergence rate of FL over the air, which theoretically quantify the impact of analog aggregation on FL. Based on the analytical results, we develop…
Exploiting Test-Time Augmentation in Federated Learning for Brain Tumor MRI Classification
Institute of Exact and Technological Sciences, Federal University of Viçosa - UFV, Rio Paranaíba-MG, Brazil（精确与技术科学研究所，弗拉维亚联邦大学-UFV，里奥帕拉纳伊巴-MG，巴西）；Department of Computing, Federal University of São Carlos, São Carlos-SP, Brazil（计算系，萨o卡洛斯联邦大学，萨o卡洛斯-SP，巴西）
Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods…
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University of Waterloo（滑铁卢大学）
Federated Learning (FL) is a prominent framework that enables training a centralized model while securing user privacy by fusing local, decentralized models. In this setting, one major obstacle is data heterogeneity, i.e., each client having non-identically and independently distributed (non-IID) data. This is analogous to the context of Domain Generalization (DG), where each client can be treated as a different domain. However, while many approaches in DG tackle data heterogeneity from the algorithmic perspective, recent evidence suggests that data augmentation can induce equal or greater per…
We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the …
Optimal Routing for Federated Learning over Dynamic Satellite Networks: Tractable or Not?
Department of Information Technology, Uppsala University（乌普萨拉大学信息科技系）；School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间利用技术与工程中心）；School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that …
School of Electrical and Computer Engineering, Ben-Gurion University of the Negev（内盖夫本-古里安大学电气与计算机工程学院）；Math and CS Faculty, Weizmann Institute of Science（魏茨曼科学研究所数学与计算机科学学院）
Federated learning (FL) is a framework for distributed learning of centralized models. In FL, a set of edge devices train a model using their local data, while repeatedly exchanging their trained updates with a central server. This procedure allows tuning a centralized model in a distributed fashion without having the users share their possibly private data. In this paper, we focus on over-the-air (OTA) FL, which has been suggested recently to reduce the communication overhead of FL due to the repeated transmissions of the model updates by a large number of users over the wireless channel. In …
Singapore University of Technology and Design（新加坡科技设计大学）；ZJU-UIUC Institute, Zhejiang University（浙江大学浙江大学-伊利诺伊大学厄巴纳香槟校区联合学院）
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. But the training efficiency is often throttled by challenges arising from limited communication and data heterogeneity. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck. Additionally, we leverage a bi-level optimization framework to personalize the…
Byzantine-Resilient Over-the-Air Federated Learning under Zero-Trust Architecture
Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）；Zhejiang Lab（之江实验室）；Zhejiang University（浙江大学）；University of California at Irvine（加州大学欧文分校）；Nanyang Technological University（南洋理工大学）
Heterogeneous Federated Learning with Convolutional and Spiking Neural Networks
College of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）；School of Engineering, Westlake University（西湖大学工程学院）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院）
Edge Intelligence Over the Air: Two Faces of Interference in Federated Learning
Singapore University of Technology and Design（新加坡科技设计大学）；Kyung Hee University（庆熙大学）；Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
BoBa: Boosting Backdoor Detection through Data Distribution Inference in Federated Learning
University of South Florida（南佛罗里达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；University of Kentucky（肯塔基大学）
Cross-device Federated Learning (FL) is a distributed learning paradigm with several challenges that differentiate it from traditional distributed learning, variability in the system characteristics on each device, and millions of clients coordinating with a central server being primary ones. Most FL systems described in the literature are synchronous - they perform a synchronized aggregation of model updates from individual clients. Scaling synchronous FL is challenging since increasing the number of clients training in parallel leads to diminishing returns in training speed, analogous to lar…
With the explosive growth of data and wireless devices, federated learning (FL) over wireless medium has emerged as a promising technology for large-scale distributed intelligent systems. Yet, the urgent demand for ubiquitous intelligence will generate a large number of concurrent FL tasks, which may seriously aggravate the scarcity of communication resources. By exploiting the analog superposition of electromagnetic waves, over-the-air computation (AirComp) is an appealing solution to alleviate the burden of communication required by FL. However, sharing frequency-time resources in over-the-a…
Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL - synchronized aggregation of client updates in FL - cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, whi…
Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping
ZJU-UIUC Institute, Zhejiang University, Haining 314400, China（浙江大学-UIUC研究院，浙江大学，海宁314400，中国）；Singapore University of Technology and Design（新加坡科技与设计大学）
Revisiting Weighted Aggregation in Federated Learning with Neural Networks
Zhejiang University（浙江大学）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；Westlake University（西湖大学）；Xiamen University（厦门大学）
Utah State University（犹他州立大学）；University of Wisconsin-Whitewater（威斯康星大学怀特沃特分校）
A new machine learning (ML) technique termed as federated learning (FL) aims to preserve data at the edge devices and to only exchange ML model parameters in the learning process. FL not only reduces the communication needs but also helps to protect the local privacy. Although FL has these advantages, it can still experience large communication latency when there are massive edge devices connected to the central parameter server (PS) and/or millions of model parameters involved in the learning process. Over-the-air computation (AirComp) is capable of computing while transmitting data by allowi…
For distributed learning among collaborative users, this paper develops and analyzes a communication-efficient scheme for federated learning (FL) over the air, which incorporates 1-bit compressive sensing (CS) into analog aggregation transmissions. To facilitate design parameter optimization, we theoretically analyze the efficacy of the proposed scheme by deriving a closed-form expression for the expected convergence rate of the FL over the air. Our theoretical results reveal the tradeoff between convergence performance and communication efficiency as a result of the aggregation errors caused …
Motivated by increasing computational capabilities of wireless devices, as well as unprecedented levels of user- and device-generated data, new distributed machine learning (ML) methods have emerged. In the wireless community, Federated Learning (FL) is of particular interest due to its communication efficiency and its ability to deal with the problem of non-IID data. FL training can be accelerated by a wireless communication method called Over-the-Air Computation (AirComp) which harnesses the interference of simultaneous uplink transmissions to efficiently aggregate model updates. However, si…
