Wireless connectivity is instrumental in enabling scalable federated learning (FL), yet wireless channels bring challenges for model training, in which channel randomness perturbs each worker's model update while multiple workers' updates incur significant interference under limited bandwidth. To address these challenges, in this work we formulate a novel constrained optimization problem, and propose an FL framework harnessing wireless channel perturbations and interference for improving privacy, bandwidth-efficiency, and scalability. The resultant algorithm is coined analog federated ADMM (A-FADMM) based on analog transmissions and the alternating direction method of multipliers (ADMM). In A-FADMM, all workers upload their model updates to the parameter server (PS) using a single channel via analog transmissions, during which all models are perturbed and aggregated over-the-air. This not only saves communication bandwidth, but also hides each worker's exact model update trajectory from any eavesdropper including the honest-but-curious PS, thereby preserving data privacy against model inversion attacks. We formally prove the convergence and privacy guarantees of A-FADMM for convex functions under time-varying channels, and numerically show the effectiveness of A-FADMM under noisy channels and stochastic non-convex functions, in terms of convergence speed and scalability, as well as communication bandwidth and energy efficiency.
Korea Advanced Institute of Science and Technology（韩国科学技术院）；King’s College London（伦敦国王学院）
Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Ga…
Class-wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory Generation
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）；College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Cyberspace, Hangzhou Dianzi University（杭州电子科技大学网络空间学院）
University of Virginia（弗吉尼亚大学）；University of Miami（迈阿密大学）；Sun Yat-Sen University（中山大学）
We advocate a new resource allocation framework, which we term resource rationing, for wireless federated learning (FL). Unlike existing resource allocation methods for FL, resource rationing focuses on balancing resources across learning rounds so that their collective impact on the federated learning performance is explicitly captured. This new framework can be integrated seamlessly with existing resource allocation schemes to optimize the convergence of FL. In particular, a novel "later-is-better" principle is at the front and center of resource rationing, which is validated empirically in …
With growth in the number of smart devices and advancements in their hardware, in recent years, data-driven machine learning techniques have drawn significant attention. However, due to privacy and communication issues, it is not possible to collect this data at a centralized location. Federated learning is a machine learning setting where the centralized location trains a learning model over remote devices. Federated learning algorithms cannot be employed in the real world scenarios unless they consider unreliable and resource-constrained nature of the wireless medium. In this paper, we propo…
There is a growing interest in the wireless communications community to complement the traditional model-based design approaches with data-driven machine learning (ML)-based solutions. While conventional ML approaches rely on the assumption of having the data and processing heads in a central entity, this is not always feasible in wireless communications applications because of the inaccessibility of private data and large communication overhead required to transmit raw data to central ML processors. As a result, decentralized ML approaches that keep the data where it is generated are much mor…
Towards Scalable Wireless Federated Learning: Challenges and Solutions
ShanghaiTech University（上海科技大学）；Nanjing University（南京大学）；Beihang University（北京航空航天大学）；Xiameng University（厦门大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
In this paper, we study a federated learning system at the wireless edge that uses over-the-air computation (AirComp). In such a system, users transmit their messages over a multi-access channel concurrently to achieve fast model aggregation. Recently, an AirComp scheme based on digital modulation has been proposed featuring one-bit gradient quantization and truncated channel inversion at users and a majority-voting based decoder at the fusion center (FC). We propose an improved digital AirComp scheme to relax its requirements on the transmitters, where users perform phase correction and trans…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Conventional machine learning techniques are conducted in a centralized manner. Recently, the massive volume of generated wireless data, the privacy concerns and the increasing computing capabilities of wireless end-devices have led to the emergence of a promising decentralized solution, termed as Wireless Federated Learning (WFL). In this first of the two parts paper, we present the application of WFL in the sixth generation of wireless networks (6G), which is envisioned to be an integrated communication and computing platform. After analyzing the key concepts of WFL, we discuss the core chal…
Hitachi (China) Research & Development Cooperation（日立（中国）研发有限公司）
In federated learning (FL), devices contribute to the global training by uploading their local model updates via wireless channels. Due to limited computation and communication resources, device scheduling is crucial to the convergence rate of FL. In this paper, we propose a joint device scheduling and resource allocation policy to maximize the model accuracy within a given total training time budget for latency constrained wireless FL. A lower bound on the reciprocal of the training performance loss, in terms of the number of training rounds and the number of scheduled devices per round, is d…
In this paper, we study the problem of federated learning over a wireless channel with user sampling, modeled by a Gaussian multiple access channel, subject to central and local differential privacy (DP/LDP) constraints. It has been shown that the superposition nature of the wireless channel provides a dual benefit of bandwidth efficient gradient aggregation, in conjunction with strong DP guarantees for the users. Specifically, the central DP privacy leakage has been shown to scale as $\mathcal{O}(1/K^{1/2})$, where $K$ is the number of users. It has also been shown that user sampling coupled …
University of California, Riverside（加利福尼亚大学河滨分校）；The Ohio State University（俄亥俄州立大学）
Potential environmental impact of machine learning by large-scale wireless networks is a major challenge for the sustainability of future smart ecosystems. In this paper, we introduce sustainable machine learning in federated learning settings, using rechargeable devices that can collect energy from the ambient environment. We propose a practical federated learning framework that leverages intermittent energy arrivals for training, with provable convergence guarantees. Our framework can be applied to a wide range of machine learning settings in networked environments, including distributed and…
Federated learning (FL) offers a decentralized learning environment so that a group of clients can collaborate to train a global model at the server, while keeping their training data confidential. This paper studies how to launch over-the-air jamming attacks to disrupt the FL process when it is executed over a wireless network. As a wireless example, FL is applied to learn how to classify wireless signals collected by clients (spectrum sensors) at different locations (such as in cooperative sensing). An adversary can jam the transmissions for the local model updates from clients to the server…
Purdue University（普渡大学）；NC State University（北卡罗来纳州立大学）
Machine learning (ML) tasks are becoming ubiquitous in today's network applications. Federated learning has emerged recently as a technique for training ML models at the network edge by leveraging processing capabilities across the nodes that collect the data. There are several challenges with employing conventional federated learning in contemporary networks, due to the significant heterogeneity in compute and communication capabilities that exist across devices. To address this, we advocate a new learning paradigm called fog learning which will intelligently distribute ML model training acro…
Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis
Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Shenzhen University（深圳大学）；University of Science and Technology Beijing（北京科技大学）；Carleton University（卡尔顿大学）；The University of British Columbia（不列颠哥伦比亚大学）
In conventional federated learning (FL), differential privacy (DP) guarantees can be obtained by injecting additional noise to local model updates before transmitting to the parameter server (PS). In the wireless FL scenario, we show that the privacy of the system can be boosted by exploiting over-the-air computation (OAC) and anonymizing the transmitting devices. In OAC, devices transmit their model updates simultaneously and in an uncoded fashion, resulting in a much more efficient use of the available spectrum. We further exploit OAC to provide anonymity for the transmitting devices. The pr…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；Imperial College London（伦敦帝国理工学院）
Federated edge learning (FEEL) is a promising distributed machine learning (ML) framework to drive edge intelligence applications. However, due to the dynamic wireless environments and the resource limitations of edge devices, communication becomes a major bottleneck. In this work, we propose time-correlated sparsification with hybrid aggregation (TCS-H) for communication-efficient FEEL, which exploits jointly the power of model compression and over-the-air computation. By exploiting the temporal correlations among model parameters, we construct a global sparsification mask, which is identical…
Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程系）；Department of Engineering, King’s College London（伦敦国王学院工程系）；Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications（北京邮电大学网络系统架构与融合北京市重点实验室）；Shenzhen Research Institute of Big Data and Future Network of Intelligence Institute, the Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）智能科学与技术研究院未来网络与大数据研究院深圳分院）
In this paper, the problem of training federated learning (FL) algorithms over a realistic wireless network is studied. In particular, in the considered model, wireless users execute an FL algorithm while training their local FL models using their own data and transmitting the trained local FL models to a base station (BS) that will generate a global FL model and send it back to the users. Since all training parameters are transmitted over wireless links, the quality of the training will be affected by wireless factors such as packet errors and the availability of wireless resources. Meanwhile…
This paper studies federated learning (FL) in a classic wireless network, where learning clients share a common wireless link to a coordinating server to perform federated model training using their local data. In such wireless federated learning networks (WFLNs), optimizing the learning performance depends crucially on how clients are selected and how bandwidth is allocated among the selected clients in every learning round, as both radio and client energy resources are limited. While existing works have made some attempts to allocate the limited wireless resources to optimize FL, they focus …
We investigate a data quality-aware dynamic client selection problem for multiple federated learning (FL) services in a wireless network, where each client offers dynamic datasets for the simultaneous training of multiple FL services, and each FL service demander has to pay for the clients under constrained monetary budgets. The problem is formalized as a non-cooperative Markov game over the training rounds. A multi-agent hybrid deep reinforcement learning-based algorithm is proposed to optimize the joint client selection and payment actions, while avoiding action conflicts. Simulation results…
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学技术研究生院）
Wireless ad hoc federated learning (WAFL) is a fully decentralized collaborative machine learning framework organized by opportunistically encountered mobile nodes. Compared to conventional federated learning, WAFL performs model training by weakly synchronizing the model parameters with others, and this shows great resilience to a poisoned model injected by an attacker. In this paper, we provide our theoretical analysis of the WAFL's resilience against model poisoning attacks, by formulating the force balance between the poisoned model and the legitimate model. According to our experiments, w…
This paper studies a federated learning (FL) system, where \textit{multiple} FL services co-exist in a wireless network and share common wireless resources. It fills the void of wireless resource allocation for multiple simultaneous FL services in the existing literature. Our method designs a two-level resource allocation framework comprising \emph{intra-service} resource allocation and \emph{inter-service} resource allocation. The intra-service resource allocation problem aims to minimize the length of FL rounds by optimizing the bandwidth allocation among the clients of each FL service. Base…
Convergence-Latency-Aware Adaptive Modulation and Resource Allocation in RIS-Assisted Wireless Federated Learning
Broadband Access Network Laboratory, Shanghai Jiao Tong University（宽带接入网络实验室，上海交通大学）；School of Information Science and Engineering, Southeast University（信息科学与工程学院，东南大学）；Data61, CSIRO（数据61，联邦科学与工业研究组织）；Department of Electronic and Electrical Engineering, University College London（电子与电气工程系，伦敦大学学院）；School of Internet of Things Engineering, Jiangnan University（物联网工程学院，江南大学）
University of Hong Kong（香港大学）；University of New South Wales（新南威尔士大学）；Southern University of Science and Technology（南方科技大学）
Wireless federated learning (FL) is an emerging machine learning paradigm that trains a global parametric model from distributed datasets via wireless communications. This paper proposes a unit-modulus wireless FL (UMWFL) framework, which simultaneously uploads local model parameters and computes global model parameters via optimized phase shifting. The proposed framework avoids sophisticated baseband signal processing, leading to both low communication delays and implementation costs. A training loss bound is derived and a penalty alternating minimization (PAM) algorithm is proposed to minimi…
Efficient Wireless Federated Learning via Low-Rank Gradient Factorization
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）
Privacy-sensitive data is stored in autonomous vehicles, smart devices, or sensor nodes that can move around with making opportunistic contact with each other. Federation among such nodes was mainly discussed in the context of federated learning with a centralized mechanism in many works. However, because of multi-vendor issues, those nodes do not want to rely on a specific server operated by a third party for this purpose. In this paper, we propose a wireless ad hoc federated learning (WAFL) -- a fully distributed cooperative machine learning organized by the nodes physically nearby. WAFL can…
Arizona State University（亚利桑那州立大学）；Harvard University（哈佛大学）
We consider a many-to-one wireless architecture for federated learning at the network edge, where multiple edge devices collaboratively train a model using local data. The unreliable nature of wireless connectivity, together with constraints in computing resources at edge devices, dictates that the local updates at edge devices should be carefully crafted and compressed to match the wireless communication resources available and should work in concert with the receiver. Thus motivated, we propose SGD-based bandlimited coordinate descent algorithms for such settings. Specifically, for the wirel…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Intel Labs（英特尔研究院）；University of Central Florida（中佛罗里达大学）
Federated learning (FL) is a distributed machine learning technology for next-generation AI systems that allows a number of workers, i.e., edge devices, collaboratively learn a shared global model while keeping their data locally to prevent privacy leakage. Enabling FL over wireless multi-hop networks can democratize AI and make it accessible in a cost-effective manner. However, the noisy bandwidth-limited multi-hop wireless connections can lead to delayed and nomadic model updates, which significantly slows down the FL convergence speed. To address such challenges, this paper aims to accelera…
Federated Learning (FL) has evolved as a promising technique to handle distributed machine learning across edge devices. A single neural network (NN) that optimises a global objective is generally learned in most work in FL, which could be suboptimal for edge devices. Although works finding a NN personalised for edge device specific tasks exist, they lack generalisation and/or convergence guarantees. In this paper, a novel communication efficient FL algorithm for personalised learning in a wireless setting with guarantees is presented. The algorithm relies on finding a ``better`` empirical est…
In this paper, we consider the problem of wireless federated learning based on sign stochastic gradient descent (signSGD) algorithm via a multiple access channel. When sending locally computed gradient's sign information, each mobile device requires to apply precoding to circumvent wireless fading effects. In practice, however, acquiring perfect knowledge of channel state information (CSI) at all mobile devices is infeasible. In this paper, we present a simple yet effective precoding method with limited channel knowledge, called sign-alignment precoding. The idea of sign-alignment precoding is…
Princeton University（普林斯顿大学）；Imperial College London（伦敦帝国理工学院）
We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD). We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its …
Singapore University of Technology and Design（新加坡科技设计大学）；Zhejiang University/University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟校区联合学院，浙江大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Electronic Science and Technology of China（电子科技大学）；The University of Exeter（埃克塞特大学）
Implementing federated learning (FL) algorithms in wireless networks has garnered a wide range of attention. However, few works have considered the impact of user mobility on the learning performance. To fill this research gap, firstly, we develop a theoretical model to characterize the hierarchical federated learning (HFL) algorithm in wireless networks where the mobile users may roam across multiple edge access points, leading to incompletion of inconsistent FL training. Secondly, we provide the convergence analysis of HFL with user mobility. Our analysis proves that the learning performance…
Mississippi State University（密西西比州立大学）；National Yang Ming Chiao Tung University（阳明交通大学）；Texas Tech University（德克萨斯理工大学）
This paper presents a novel approach to conduct highly efficient federated learning (FL) over a massive wireless edge network, where an edge server and numerous mobile devices (clients) jointly learn a global model without transporting the huge amount of data collected by the mobile devices to the edge server. The proposed FL approach is referred to as spatio-temporal FL (STFL), which jointly exploits the spatial and temporal correlations between the learning updates from different mobile devices scheduled to join STFL in various training epochs. The STFL model not only represents the realisti…
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Waterloo（滑铁卢大学）
Federated learning (FL) over resource-constrained wireless networks has recently attracted much attention. However, most existing studies consider one FL task in single-cell wireless networks and ignore the impact of downlink/uplink inter-cell interference on the learning performance. In this paper, we investigate FL over a multi-cell wireless network, where each cell performs a different FL task and over-the-air computation (AirComp) is adopted to enable fast uplink gradient aggregation. We conduct convergence analysis of AirComp-assisted FL systems, taking into account the inter-cell interfe…
A Novel Collaborative Framework for Efficient Synchronization in Split Federated Learning over Wireless Networks
Department of Electrical and Computer Engineering, Concordia University（电气与计算机工程系，康科迪亚大学）；Department of Physics and Computer Science, Faculty of Science, Wilfrid Laurier University（物理与计算机科学系，科学学院，威尔弗里德·劳里埃大学）
Texas A&M University（德克萨斯农工大学）；Texas A&M University at Qatar（德克萨斯农工大学卡塔尔分校）；Qatar Foundation Research, Development and Innovation RDI（卡塔尔基金会研究、开发与创新（RDI））；Tennessee Tech University（田纳西理工大学）
This paper proposes using communication pipelining to enhance the wireless spectrum utilization efficiency and convergence speed of federated learning in mobile edge computing applications. Due to limited wireless sub-channels, a subset of the total clients is scheduled in each iteration of federated learning algorithms. On the other hand, the scheduled clients wait for the slowest client to finish its computation. We propose to first cluster the clients based on the time they need per iteration to compute the local gradients of the federated learning model. Then, we schedule a mixture of clie…
Thompson Rivers University（汤普森河大学）；University of Luxembourg（卢森堡大学）
In order to meet the extremely heterogeneous requirements of the next generation wireless communication networks, research community is increasingly dependent on using machine learning solutions for real-time decision-making and radio resource management. Traditional machine learning employs fully centralized architecture in which the entire training data is collected at one node e.g., cloud server, that significantly increases the communication overheads and also raises severe privacy concerns. Towards this end, a distributed machine learning paradigm termed as Federated learning (FL) has bee…
Federated learning (FL) has recently emerged as an attractive decentralized solution for wireless networks to collaboratively train a shared model while keeping data localized. As a general approach, existing FL methods tend to assume perfect knowledge of the Channel State Information (CSI) during the training phase, which may not be easy to acquire in case of fast fading channels. Moreover, literature analyses either consider a fixed number of clients participating in the training of the federated model, or simply assume that all clients operate at the maximum achievable rate to transmit mode…
Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Sampling
Beijing University of Posts and Telecommunications（北京邮电大学）；Hong Kong Baptist University（香港浸会大学）；Peng Cheng Laboratory（鹏城实验室）；Duke Kunshan University（昆山杜克大学）
University of Houston（休斯顿大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology of China（中国科学技术大学）；University of Florida（佛罗里达大学）
Federated learning (FL) is a popular collaborative distributed machine learning paradigm across mobile devices. However, practical FL over resource constrained mobile devices confronts multiple challenges, e.g., the local on-device training and model updates in FL are power hungry and radio resource intensive for mobile devices. To address these challenges, in this paper, we attempt to take FL into the design of future wireless networks and develop a novel joint design of wireless transmission and weight quantization for energy efficient FL over mobile devices. Specifically, we develop flexibl…
