V2: Corrected typos and some technical content. Added some references. Added some theoretical proofs. Added some content to make our research applicable to more scenarios
Federated learning (FL) is an attractive paradigm for making use of rich distributed data while protecting data privacy. Nonetheless, nonideal communication links and limited transmission resources may hinder the implementation of fast and accurate FL. In this paper, we study joint optimization of communications and FL based on analog aggregation transmission in realistic wireless networks. We first derive closed-form expressions for the expected convergence rate of FL over the air, which theoretically quantify the impact of analog aggregation on FL. Based on the analytical results, we develop a joint optimization model for accurate FL implementation, which allows a parameter server to select a subset of workers and determine an appropriate power scaling factor. Since the practical setting of FL over the air encounters unobservable parameters, we reformulate the joint optimization of worker selection and power allocation using controlled approximation. Finally, we efficiently solve the resulting mixed-integer programming problem via a simple yet optimal finite-set search method by reducing the search space. Simulation results show that the proposed solutions developed for realistic wireless analog channels outperform a benchmark method, and achieve comparable performance of the ideal case where FL is implemented over noise-free wireless channels.
Update Estimation and Scheduling for Over-the-Air Federated Learning with Energy Harvesting Devices
Dept. of Electrical and Electronics Engineering, Bilkent University（电子与电子工程系，比尔肯特大学）；Dept. of Electrical and Electronic Engineering, Imperial College London（电子与电子工程系，帝国理工学院伦敦分校）
This paper investigates the robustness of over-the-air federated learning to Byzantine attacks. The simple averaging of the model updates via over-the-air computation makes the learning task vulnerable to random or intended modifications of the local model updates of some malicious clients. We propose a robust transmission and aggregation framework to such attacks while preserving the benefits of over-the-air computation for federated learning. For the proposed robust federated learning, the participating clients are randomly divided into groups and a transmission time slot is allocated to eac…
This paper considers over-the-air federated learning (OTA-FL). OTA-FL exploits the superposition property of the wireless medium, and performs model aggregation over the air for free. Thus, it can greatly reduce the communication cost incurred in communicating model updates from the edge devices. In order to fully utilize this advantage while providing comparable learning performance to conventional federated learning that presumes model aggregation via noiseless channels, we consider the joint design of transmission scaling and the number of local iterations at each round, given the power con…
Edge Intelligence Over the Air: Two Faces of Interference in Federated Learning
Singapore University of Technology and Design（新加坡科技设计大学）；Kyung Hee University（庆熙大学）；Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Singapore University of Technology and Design（新加坡科技设计大学）；ZJU-UIUC Institute, Zhejiang University（浙江大学浙江大学-伊利诺伊大学厄巴纳香槟校区联合学院）
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. But the training efficiency is often throttled by challenges arising from limited communication and data heterogeneity. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck. Additionally, we leverage a bi-level optimization framework to personalize the…
AirPASS: Over-the-Air Federated Learning via Pinching Antenna Systems
Department of Electrical Engineering, Chalmers University of Technology（楚德大学电气工程系）；School of Electrical and Computer Engineering, Purdue University（普渡大学电气与计算机工程学院）
In conventional federated learning (FL), differential privacy (DP) guarantees can be obtained by injecting additional noise to local model updates before transmitting to the parameter server (PS). In the wireless FL scenario, we show that the privacy of the system can be boosted by exploiting over-the-air computation (OAC) and anonymizing the transmitting devices. In OAC, devices transmit their model updates simultaneously and in an uncoded fashion, resulting in a much more efficient use of the available spectrum. We further exploit OAC to provide anonymity for the transmitting devices. The pr…
For distributed learning among collaborative users, this paper develops and analyzes a communication-efficient scheme for federated learning (FL) over the air, which incorporates 1-bit compressive sensing (CS) into analog aggregation transmissions. To facilitate design parameter optimization, we theoretically analyze the efficacy of the proposed scheme by deriving a closed-form expression for the expected convergence rate of the FL over the air. Our theoretical results reveal the tradeoff between convergence performance and communication efficiency as a result of the aggregation errors caused …
From Federated Learning to Quantum Federated Learning for Space-Air-Ground Integrated Networks
Hung Yen University of Technology and Education（兴安技术与教育大学）；University of Tasmania（塔斯马尼亚大学）；University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；Lac Hong University（乐鸿大学）；Manchester Metropolitan University（曼彻斯特城市大学）；International Hellenic University（国际希腊大学）
In this paper, we consider the problem of wireless federated learning based on sign stochastic gradient descent (signSGD) algorithm via a multiple access channel. When sending locally computed gradient's sign information, each mobile device requires to apply precoding to circumvent wireless fading effects. In practice, however, acquiring perfect knowledge of channel state information (CSI) at all mobile devices is infeasible. In this paper, we present a simple yet effective precoding method with limited channel knowledge, called sign-alignment precoding. The idea of sign-alignment precoding is…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
In this paper, we study a federated learning system at the wireless edge that uses over-the-air computation (AirComp). In such a system, users transmit their messages over a multi-access channel concurrently to achieve fast model aggregation. Recently, an AirComp scheme based on digital modulation has been proposed featuring one-bit gradient quantization and truncated channel inversion at users and a majority-voting based decoder at the fusion center (FC). We propose an improved digital AirComp scheme to relax its requirements on the transmitters, where users perform phase correction and trans…
Heilongjiang University（黑龙江大学）；Nanyang Technological University（南洋理工大学）；Georgia Southern University（佐治亚南方大学）；Alibaba Group（阿里巴巴集团）；Alibaba-NTU Joint Research Institute（阿里巴巴-南洋理工大学联合研究院）；Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly（南洋理工大学-不列颠哥伦比亚大学积极乐龄卓越联合研究中心）
Due to air quality significantly affects human health, it is becoming increasingly important to accurately and timely predict the Air Quality Index (AQI). To this end, this paper proposes a new federated learning-based aerial-ground air quality sensing framework for fine-grained 3D air quality monitoring and forecasting. Specifically, in the air, this framework leverages a light-weight Dense-MobileNet model to achieve energy-efficient end-to-end learning from haze features of haze images taken by Unmanned Aerial Vehicles (UAVs) for predicting AQI scale distribution. Furthermore, the Federated …
Over-the-Air Federated Learning: Rethinking Edge AI Through Signal Processing
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（皇家理工学院电气工程与计算机科学学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）
This paper develops a class of low-complexity device scheduling algorithms for over-the-air federated learning via the method of matching pursuit. The proposed scheme tracks closely the close-to-optimal performance achieved by difference-of-convex programming, and outperforms significantly the well-known benchmark algorithms based on convex relaxation. Compared to the state-of-the-art, the proposed scheme poses a drastically lower computational load on the system: For $K$ devices and $N$ antennas at the parameter server, the benchmark complexity scales with $\left(N^2+K\right)^3 + N^6$ while t…
Motivated by increasing computational capabilities of wireless devices, as well as unprecedented levels of user- and device-generated data, new distributed machine learning (ML) methods have emerged. In the wireless community, Federated Learning (FL) is of particular interest due to its communication efficiency and its ability to deal with the problem of non-IID data. FL training can be accelerated by a wireless communication method called Over-the-Air Computation (AirComp) which harnesses the interference of simultaneous uplink transmissions to efficiently aggregate model updates. However, si…
NCAirFL: CSI-Free Over-the-Air Federated Learning Based on Non-Coherent Detection
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Arizona State University（亚利桑那州立大学）；The Hong Kong University of Science and Technology（香港科技大学）
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we consider decentralized federated learning (FL) over wireless networks, where over-the-air computation (AirComp) is adopted to facilitate the local model consensus in a device-to-device (D2D) communication manner. However, the AirComp-based consensus phase brings the additive noise in each algorithm iterate and the consensus needs to be robust to wireless network topology changes, which introduce a coupled and novel challenge of establishing the convergence for wireless decentralized FL algorithm. To facilitate consensus phase, we propose an AirComp-based DSGD with gradient tr…
Utah State University（犹他州立大学）；University of Wisconsin-Whitewater（威斯康星大学怀特沃特分校）
A new machine learning (ML) technique termed as federated learning (FL) aims to preserve data at the edge devices and to only exchange ML model parameters in the learning process. FL not only reduces the communication needs but also helps to protect the local privacy. Although FL has these advantages, it can still experience large communication latency when there are massive edge devices connected to the central parameter server (PS) and/or millions of model parameters involved in the learning process. Over-the-air computation (AirComp) is capable of computing while transmitting data by allowi…
Rethinking Federated Learning Over the Air: The Blessing of Scaling Up
Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学/伊利诺伊大学厄巴纳-香槟分校研究所）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Engineering Systems and Design, Singapore University of Technology and Design（新加坡科技与设计大学工程系统与设计系）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机科学学院）；Department of Computer and Information Science, Linköping University（_linköping大学计算机与信息科学系）
With the explosive growth of data and wireless devices, federated learning (FL) over wireless medium has emerged as a promising technology for large-scale distributed intelligent systems. Yet, the urgent demand for ubiquitous intelligence will generate a large number of concurrent FL tasks, which may seriously aggravate the scarcity of communication resources. By exploiting the analog superposition of electromagnetic waves, over-the-air computation (AirComp) is an appealing solution to alleviate the burden of communication required by FL. However, sharing frequency-time resources in over-the-a…
Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping
ZJU-UIUC Institute, Zhejiang University, Haining 314400, China（浙江大学-UIUC研究院，浙江大学，海宁314400，中国）；Singapore University of Technology and Design（新加坡科技与设计大学）
Federated learning (FL) is a promising technique that enables many edge devices to train a machine learning model collaboratively in wireless networks. By exploiting the superposition nature of wireless waveforms, over-the-air computation (AirComp) can accelerate model aggregation and hence facilitate communication-efficient FL. Due to channel fading, power control is crucial in AirComp. Prior works assume that the signals to be aggregated from each device, i.e., local gradients have identical statistics. In FL, however, gradient statistics vary over both training iterations and feature dimens…
Beijing University of Posts and Telecommunications（北京邮电大学）；Commonwealth Scientific and Industrial Research Organization（联邦科学与工业研究组织）
Over-the-air federated learning (AirFL) allows devices to train a learning model in parallel and synchronize their local models using over-the-air computation. The integrity of AirFL is vulnerable due to the obscurity of the local models aggregated over-the-air. This paper presents a novel framework to balance the accuracy and integrity of AirFL, where multi-antenna devices and base station (BS) are jointly optimized with a reconfigurable intelligent surface (RIS). The key contributions include a new and non-trivial problem jointly considering the model accuracy and integrity of AirFL, and a n…
Byzantine-Resilient Over-the-Air Federated Learning under Zero-Trust Architecture
Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）；Zhejiang Lab（之江实验室）；Zhejiang University（浙江大学）；University of California at Irvine（加州大学欧文分校）；Nanyang Technological University（南洋理工大学）
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Imperial College London（伦敦帝国理工学院）
Federated edge learning (FEEL) is a promising distributed machine learning (ML) framework to drive edge intelligence applications. However, due to the dynamic wireless environments and the resource limitations of edge devices, communication becomes a major bottleneck. In this work, we propose time-correlated sparsification with hybrid aggregation (TCS-H) for communication-efficient FEEL, which exploits jointly the power of model compression and over-the-air computation. By exploiting the temporal correlations among model parameters, we construct a global sparsification mask, which is identical…
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Houston（休斯顿大学）；Louisiana State University（路易斯安那州立大学）；Shandong Artificial Intelligence Institute（山东省人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））
Federated learning (FL) enables mobile devices to collaboratively learn a shared prediction model while keeping data locally. However, there are two major research challenges to practically deploy FL over mobile devices: (i) frequent wireless updates of huge size gradients v.s. limited spectrum resources, and (ii) energy-hungry FL communication and local computing during training v.s. battery-constrained mobile devices. To address those challenges, in this paper, we propose a novel multi-bit over-the-air computation (M-AirComp) approach for spectrum-efficient aggregation of local model updates…
Federated learning framework for collaborative remaining useful life prognostics: an aircraft engine case study
Department of Information and Computing Sciences, Utrecht University（乌特勒支大学信息与计算科学系）；Faculty of Aerospace Engineering, Delft University of Technology（代尔夫特理工大学航空航天工程学院）
Communication-Learning Co-Design for Differentially Private Over-the-Air Federated Distillation
Department of Information Engineering, The Chinese University of Hong Kong（信息工程系，香港中文大学）；Intelligent Transportation Thrust, The Hong Kong University of Science and Technology (Guangzhou)（智能交通研究组，香港科学与技术大学（广州））
East China Normal University（华东师范大学）；Software Engineering Institute（软件工程研究所）；EPFL（洛桑联邦理工学院）；ShanghaiTech University（上海科技大学）
Federated learning (FL) is a promising learning paradigm that can tackle the increasingly prominent isolated data islands problem while keeping users' data locally with privacy and security guarantees. However, FL could result in task-oriented data traffic flows over wireless networks with limited radio resources. To design communication-efficient FL, most of the existing studies employ the first-order federated optimization approach that has a slow convergence rate. This however results in excessive communication rounds for local model updates between the edge devices and edge server. To addr…
We explore a scheme that enables the training of a deep neural network in a Federated Learning configuration over an additive white Gaussian noise channel. The goal is to create a low complexity, linear compression strategy, called PolarAir, that reduces the size of the gradient at the user side to lower the number of channel uses needed to transmit it. The suggested approach belongs to the family of compressed sensing techniques, yet it constructs the sensing matrix and the recovery procedure using multiple access techniques. Simulations show that it can reduce the number of channel uses by ~…
