Internet of Things (IoT) services will use machine learning tools to efficiently analyze various types of data collected by IoT devices for inference, autonomy, and control purposes. However, due to resource constraints and privacy challenges, edge IoT devices may not be able to transmit their collected data to a central controller for training machine learning models. To overcome this challenge, federated learning (FL) has been proposed as a means for enabling edge devices to train a shared machine learning model without data exchanges thus reducing communication overhead and preserving data privacy. However, Google's seminal FL algorithm requires all devices to be directly connected with a central controller, which significantly limits its application scenarios. In this context, this paper introduces a novel FL framework, called collaborative FL (CFL), which enables edge devices to implement FL with less reliance on a central controller. The fundamentals of this framework are developed and then, a number of communication techniques are proposed so as to improve the performance of CFL. To this end, an overview of centralized learning, Google's seminal FL, and CFL is first presented. For each type of learning, the basic architecture as well as its advantages, drawbacks, and usage conditions are introduced. Then, three CFL performance metrics are presented and a suite of communication techniques ranging from network formation, device scheduling, mobility management, and coding is introduced to optimize the performance of CFL. For each technique, future research opportunities are also discussed. In a nutshell, this article will showcase how the proposed CFL framework can be effectively implemented at the edge of large-scale wireless systems such as the Internet of Things.
Korea Advanced Institute of Science and Technology（韩国科学技术院）；King’s College London（伦敦国王学院）
Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Ga…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
In this paper, we study a federated learning system at the wireless edge that uses over-the-air computation (AirComp). In such a system, users transmit their messages over a multi-access channel concurrently to achieve fast model aggregation. Recently, an AirComp scheme based on digital modulation has been proposed featuring one-bit gradient quantization and truncated channel inversion at users and a majority-voting based decoder at the fusion center (FC). We propose an improved digital AirComp scheme to relax its requirements on the transmitters, where users perform phase correction and trans…
University of California, Riverside（加利福尼亚大学河滨分校）；The Ohio State University（俄亥俄州立大学）
Potential environmental impact of machine learning by large-scale wireless networks is a major challenge for the sustainability of future smart ecosystems. In this paper, we introduce sustainable machine learning in federated learning settings, using rechargeable devices that can collect energy from the ambient environment. We propose a practical federated learning framework that leverages intermittent energy arrivals for training, with provable convergence guarantees. Our framework can be applied to a wide range of machine learning settings in networked environments, including distributed and…
University of Virginia（弗吉尼亚大学）；University of Miami（迈阿密大学）；Sun Yat-Sen University（中山大学）
We advocate a new resource allocation framework, which we term resource rationing, for wireless federated learning (FL). Unlike existing resource allocation methods for FL, resource rationing focuses on balancing resources across learning rounds so that their collective impact on the federated learning performance is explicitly captured. This new framework can be integrated seamlessly with existing resource allocation schemes to optimize the convergence of FL. In particular, a novel "later-is-better" principle is at the front and center of resource rationing, which is validated empirically in …
National University of Singapore（新加坡国立大学）；Wuhan University（武汉大学）
In current deep learning paradigms, local training or the Standalone framework tends to result in overfitting and thus poor generalizability. This problem can be addressed by Distributed or Federated Learning (FL) that leverages a parameter server to aggregate model updates from individual participants. However, most existing Distributed or FL frameworks have overlooked an important aspect of participation: collaborative fairness. In particular, all participants can receive the same or similar models, regardless of their contributions. To address this issue, we investigate the collaborative fa…
In this paper, we study the problem of federated learning over a wireless channel with user sampling, modeled by a Gaussian multiple access channel, subject to central and local differential privacy (DP/LDP) constraints. It has been shown that the superposition nature of the wireless channel provides a dual benefit of bandwidth efficient gradient aggregation, in conjunction with strong DP guarantees for the users. Specifically, the central DP privacy leakage has been shown to scale as $\mathcal{O}(1/K^{1/2})$, where $K$ is the number of users. It has also been shown that user sampling coupled …
Universidade Federal Fluminense（弗鲁米嫩塞联邦大学）；Trinity College Dublin（都柏林圣三一学院）
Fast identification of new network attack patterns is crucial for improving network security. Nevertheless, identifying an ongoing attack in a heterogeneous network is a non-trivial task. Federated learning emerges as a solution to collaborative training for an Intrusion Detection System (IDS). The federated learning-based IDS trains a global model using local machine learning models provided by federated participants without sharing local data. However, optimization challenges are intrinsic to federated learning. This paper proposes the Federated Simulated Annealing (FedSA) metaheuristic to s…
With the fast development of quantum computing and deep learning, quantum neural networks have attracted great attention recently. By leveraging the power of quantum computing, deep neural networks can potentially overcome computational power limitations in classic machine learning. However, when multiple quantum machines wish to train a global model using the local data on each machine, it may be very difficult to copy the data into one machine and train the model. Therefore, a collaborative quantum neural network framework is necessary. In this article, we borrow the core idea of federated l…
Beijing University of Posts and Telecommunications（北京邮电大学）
Implementing existing federated learning in massive Internet of Things (IoT) networks faces critical challenges such as imbalanced and statistically heterogeneous data and device diversity. To this end, we propose a semi-federated learning (SemiFL) framework to provide a potential solution for the realization of intelligent IoT. By seamlessly integrating the centralized and federated paradigms, our SemiFL framework shows high scalability in terms of the number of IoT devices even in the presence of computing-limited sensors. Furthermore, compared to traditional learning approaches, the propose…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Conventional machine learning techniques are conducted in a centralized manner. Recently, the massive volume of generated wireless data, the privacy concerns and the increasing computing capabilities of wireless end-devices have led to the emergence of a promising decentralized solution, termed as Wireless Federated Learning (WFL). In this first of the two parts paper, we present the application of WFL in the sixth generation of wireless networks (6G), which is envisioned to be an integrated communication and computing platform. After analyzing the key concepts of WFL, we discuss the core chal…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）
With the deployment of smart sensors and advancements in communication technologies, big data analytics have become vastly popular in the smart grid domain, informing stakeholders of the best power utilization strategy. However, these power-related data are stored and owned by different parties. For example, power consumption data are stored in numerous transformer stations across cities; mobility data of the population, which are important indicators of power consumption, are held by mobile companies. Direct data sharing might compromise party benefits, individual privacy and even national se…
Towards Scalable Wireless Federated Learning: Challenges and Solutions
ShanghaiTech University（上海科技大学）；Nanjing University（南京大学）；Beihang University（北京航空航天大学）；Xiameng University（厦门大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Wireless connectivity is instrumental in enabling scalable federated learning (FL), yet wireless channels bring challenges for model training, in which channel randomness perturbs each worker's model update while multiple workers' updates incur significant interference under limited bandwidth. To address these challenges, in this work we formulate a novel constrained optimization problem, and propose an FL framework harnessing wireless channel perturbations and interference for improving privacy, bandwidth-efficiency, and scalability. The resultant algorithm is coined analog federated ADMM (A-…
There is a growing interest in the wireless communications community to complement the traditional model-based design approaches with data-driven machine learning (ML)-based solutions. While conventional ML approaches rely on the assumption of having the data and processing heads in a central entity, this is not always feasible in wireless communications applications because of the inaccessibility of private data and large communication overhead required to transmit raw data to central ML processors. As a result, decentralized ML approaches that keep the data where it is generated are much mor…
With growth in the number of smart devices and advancements in their hardware, in recent years, data-driven machine learning techniques have drawn significant attention. However, due to privacy and communication issues, it is not possible to collect this data at a centralized location. Federated learning is a machine learning setting where the centralized location trains a learning model over remote devices. Federated learning algorithms cannot be employed in the real world scenarios unless they consider unreliable and resource-constrained nature of the wireless medium. In this paper, we propo…
In conventional federated learning (FL), differential privacy (DP) guarantees can be obtained by injecting additional noise to local model updates before transmitting to the parameter server (PS). In the wireless FL scenario, we show that the privacy of the system can be boosted by exploiting over-the-air computation (OAC) and anonymizing the transmitting devices. In OAC, devices transmit their model updates simultaneously and in an uncoded fashion, resulting in a much more efficient use of the available spectrum. We further exploit OAC to provide anonymity for the transmitting devices. The pr…
Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Imperial College London（伦敦帝国理工学院）
Intermittent connectivity of clients to the parameter server (PS) is a major bottleneck in federated edge learning frameworks. The lack of constant connectivity induces a large generalization gap, especially when the local data distribution amongst clients exhibits heterogeneity. To overcome intermittent communication outages between clients and the central PS, we introduce the concept of collaborative relaying wherein the participating clients relay their neighbors' local updates to the PS in order to boost the participation of clients with poor connectivity to the PS. We propose a semi-decen…
This paper studies federated learning (FL) in a classic wireless network, where learning clients share a common wireless link to a coordinating server to perform federated model training using their local data. In such wireless federated learning networks (WFLNs), optimizing the learning performance depends crucially on how clients are selected and how bandwidth is allocated among the selected clients in every learning round, as both radio and client energy resources are limited. While existing works have made some attempts to allocate the limited wireless resources to optimize FL, they focus …
A Novel Collaborative Framework for Efficient Synchronization in Split Federated Learning over Wireless Networks
Department of Electrical and Computer Engineering, Concordia University（电气与计算机工程系，康科迪亚大学）；Department of Physics and Computer Science, Faculty of Science, Wilfrid Laurier University（物理与计算机科学系，科学学院，威尔弗里德·劳里埃大学）
Current content filtering and blocking methods are susceptible to various circumvention techniques and are relatively slow in dealing with new threats. This is due to these methods using shallow pattern recognition that is based on regular expression rules found in crowdsourced block lists. We propose a novel system that aims to remedy the aforementioned issues by examining deep textual patterns of network-oriented content relating to the domain being interacted with. Moreover, we propose to use federated learning that allows users to take advantage of each other's localized knowledge/experien…
We investigate a data quality-aware dynamic client selection problem for multiple federated learning (FL) services in a wireless network, where each client offers dynamic datasets for the simultaneous training of multiple FL services, and each FL service demander has to pay for the clients under constrained monetary budgets. The problem is formalized as a non-cooperative Markov game over the training rounds. A multi-agent hybrid deep reinforcement learning-based algorithm is proposed to optimize the joint client selection and payment actions, while avoiding action conflicts. Simulation results…
Due to the different losses caused by various photovoltaic (PV) array faults, accurate diagnosis of fault types is becoming increasingly important. Compared with a single one, multiple PV stations collect sufficient fault samples, but their data is not allowed to be shared directly due to potential conflicts of interest. Therefore, federated learning can be exploited to train a collaborative fault diagnosis model. However, the modeling efficiency is seriously affected by the model update mechanism since each PV station has a different computing capability and amount of data. Moreover, for the …
University of Hong Kong（香港大学）；University of New South Wales（新南威尔士大学）；Southern University of Science and Technology（南方科技大学）
Wireless federated learning (FL) is an emerging machine learning paradigm that trains a global parametric model from distributed datasets via wireless communications. This paper proposes a unit-modulus wireless FL (UMWFL) framework, which simultaneously uploads local model parameters and computes global model parameters via optimized phase shifting. The proposed framework avoids sophisticated baseband signal processing, leading to both low communication delays and implementation costs. A training loss bound is derived and a penalty alternating minimization (PAM) algorithm is proposed to minimi…
University of Singapore（新加坡大学）；Ant financial（蚂蚁金融）
Federated learning (FL) is an emerging practical framework for effective and scalable machine learning among multiple participants, such as end users, organizations and companies. However, most existing FL or distributed learning frameworks have not well addressed two important issues together: collaborative fairness and adversarial robustness (e.g. free-riders and malicious participants). In conventional FL, all participants receive the global model (equal rewards), which might be unfair to the high-contributing participants. Furthermore, due to the lack of a safeguard mechanism, free-riders …
Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Imperial College London（伦敦帝国理工学院）
We present a semi-decentralized federated learning algorithm wherein clients collaborate by relaying their neighbors' local updates to a central parameter server (PS). At every communication round to the PS, each client computes a local consensus of the updates from its neighboring clients and eventually transmits a weighted average of its own update and those of its neighbors to the PS. We appropriately optimize these averaging weights to ensure that the global update at the PS is unbiased and to reduce the variance of the global update at the PS, consequently improving the rate of convergenc…
Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Serving
Department of Electrical Engineering and Computer Science, University of Stavanger（电子工程与计算机科学系，斯塔万格大学）；Department of Data Science, The Catholic University of Korea（数据科学系，韩国天主大学）
Federated Learning via Input-Output Collaborative Distillation
University at Buffalo（布法罗大学）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Harvard Medical School（哈佛医学院）；Virginia Tech（弗吉尼亚理工大学）；Jarvis Research Center, Tencent YouTu Lab（腾讯优图实验室杰瑞斯研究中心）；United Imaging Intelligence（联影智能）；Hangzhou Research Institute, Beihang University（北京航空航天大学杭州研究院）；Zhongguancun Laboratory（中关村实验室）；Nanchang Institute of Technology（南昌工学院）
Texas A&M University（德克萨斯农工大学）；Texas A&M University at Qatar（德克萨斯农工大学卡塔尔分校）；Qatar Foundation Research, Development and Innovation RDI（卡塔尔基金会研究、开发与创新（RDI））；Tennessee Tech University（田纳西理工大学）
This paper proposes using communication pipelining to enhance the wireless spectrum utilization efficiency and convergence speed of federated learning in mobile edge computing applications. Due to limited wireless sub-channels, a subset of the total clients is scheduled in each iteration of federated learning algorithms. On the other hand, the scheduled clients wait for the slowest client to finish its computation. We propose to first cluster the clients based on the time they need per iteration to compute the local gradients of the federated learning model. Then, we schedule a mixture of clie…
In this paper, we consider the problem of wireless federated learning based on sign stochastic gradient descent (signSGD) algorithm via a multiple access channel. When sending locally computed gradient's sign information, each mobile device requires to apply precoding to circumvent wireless fading effects. In practice, however, acquiring perfect knowledge of channel state information (CSI) at all mobile devices is infeasible. In this paper, we present a simple yet effective precoding method with limited channel knowledge, called sign-alignment precoding. The idea of sign-alignment precoding is…
Federated learning (FL) offers a decentralized learning environment so that a group of clients can collaborate to train a global model at the server, while keeping their training data confidential. This paper studies how to launch over-the-air jamming attacks to disrupt the FL process when it is executed over a wireless network. As a wireless example, FL is applied to learn how to classify wireless signals collected by clients (spectrum sensors) at different locations (such as in cooperative sensing). An adversary can jam the transmissions for the local model updates from clients to the server…
Federated learning (FL) has recently emerged as an attractive decentralized solution for wireless networks to collaboratively train a shared model while keeping data localized. As a general approach, existing FL methods tend to assume perfect knowledge of the Channel State Information (CSI) during the training phase, which may not be easy to acquire in case of fast fading channels. Moreover, literature analyses either consider a fixed number of clients participating in the training of the federated model, or simply assume that all clients operate at the maximum achievable rate to transmit mode…
George Mason University（乔治梅森大学）；Microsoft（微软）；University of Maryland, Baltimore County（马里兰大学巴尔的摩郡分校）
Federated learning learns from scattered data by fusing collaborative models from local nodes. However, due to chaotic information distribution, the model fusion may suffer from structural misalignment with regard to unmatched parameters. In this work, we propose a novel federated learning framework to resolve this issue by establishing a firm structure-information alignment across collaborative models. Specifically, we design a feature-oriented regulation method ({$Ψ$-Net}) to ensure explicit feature information allocation in different neural network structures. Applying this regulating metho…
Semantic Communication in Dynamic Channel Scenarios: Collaborative Optimization of Dual-Pipeline Joint Source-Channel Coding and Personalized Federated Learning
Beijing Institute of Technology（北京理工大学）；Institute of Computer Application Technology Norinco Groups（中国兵器工业集团计算机应用技术研究所）
The continuous strengthening of the security posture of IoT ecosystems is vital due to the increasing number of interconnected devices and the volume of sensitive data shared. The utilisation of Machine Learning (ML) capabilities in the defence against IoT cyber attacks has many potential benefits. However, the currently proposed frameworks do not consider data privacy, secure architectures, and/or scalable deployments of IoT ecosystems. In this paper, we propose a hierarchical blockchain-based federated learning framework to enable secure and privacy-preserved collaborative IoT intrusion dete…
Qingdao University（青岛大学）；Jilin University（吉林大学）；Kennesaw State University（肯尼索州立大学）；Georgia State University（佐治亚州立大学）
In this work, we propose a collaborative city digital twin based on FL, a novel paradigm that allowing multiple city DT to share the local strategy and status in a timely manner. In particular, an FL central server manages the local updates of multiple collaborators (city DT), provides a global model which is trained in multiple iterations at different city DT systems, until the model gains the correlations between various response plan and infection trend. That means, a collaborative city DT paradigm based on FL techniques can obtain knowledge and patterns from multiple DTs, and eventually es…
JD Finance America Corporation（京东金融美国公司）；Department of Electrical & Computer Engineering, University of Pittsburgh（匹兹堡大学电气与计算机工程系）；University of Pittsburgh（匹兹堡大学）
The privacy-preserving federated learning for vertically partitioned data has shown promising results as the solution of the emerging multi-party joint modeling application, in which the data holders (such as government branches, private finance and e-business companies) collaborate throughout the learning process rather than relying on a trusted third party to hold data. However, existing federated learning algorithms for vertically partitioned data are limited to synchronous computation. To improve the efficiency when the unbalanced computation/communication resources are common among the pa…
Thompson Rivers University（汤普森河大学）；University of Luxembourg（卢森堡大学）
In order to meet the extremely heterogeneous requirements of the next generation wireless communication networks, research community is increasingly dependent on using machine learning solutions for real-time decision-making and radio resource management. Traditional machine learning employs fully centralized architecture in which the entire training data is collected at one node e.g., cloud server, that significantly increases the communication overheads and also raises severe privacy concerns. Towards this end, a distributed machine learning paradigm termed as Federated learning (FL) has bee…
When Differential Privacy Meets Wireless Federated Learning: An Improved Analysis for Privacy and Convergence
School of Informatics, Xiamen University, China（厦门大学信息学院，中国）；Internet of Things Thrust, Hong Kong University of Science and Technology (Guangzhou), China（香港科技大学（广州）物联网研究组，中国）
George Mason University（乔治梅森大学）；Microsoft（微软）；University of Maryland, Baltimore County（马里兰大学巴尔的摩分校）
Federated learning learns from scattered data by fusing collaborative models from local nodes. However, the conventional coordinate-based model averaging by FedAvg ignored the random information encoded per parameter and may suffer from structural feature misalignment. In this work, we propose Fed2, a feature-aligned federated learning framework to resolve this issue by establishing a firm structure-feature alignment across the collaborative models. Fed2 is composed of two major designs: First, we design a feature-oriented model structure adaptation method to ensure explicit feature allocation…
Being very low power, the use of neuromorphic processors in mobile devices to solve machine learning problems is a promising alternative to traditional Von Neumann processors. Federated Learning enables entities such as mobile devices to collaboratively learn a shared model while keeping their training data local. Additionally, federated learning is a secure way of learning because only the model weights need to be shared between models, keeping the data private. Here we demonstrate the efficacy of federated learning in neuromorphic processors. Neuromorphic processors benefit from the collabor…
