Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
Princeton University（普林斯顿大学）；Imperial College London（帝国理工学院）；The University of Hong Kong（香港大学）；Virginia Tech（弗吉尼亚理工大学）；University of Oulu（奥卢大学）；Ericsson Research（爱立信研究院）
Mingzhe Chen, Deniz Gündüz, Kaibin Huang, Walid Saad, Mehdi Bennis, Aneta Vulgarakis Feljan, H. Vincent Poor
The next-generation of wireless networks will enable many machine learning (ML) tools and applications to efficiently analyze various types of data collected by edge devices for inference, autonomy, and decision making purposes. However, due to resource constraints, delay limitations, and privacy challenges, edge devices cannot offload their entire collected datasets to a cloud server for centrally training their ML models or inference purposes. To overcome these challenges, distributed learning and inference techniques have been proposed as a means to enable edge devices to collaboratively train ML models without raw data exchanges, thus reducing the communication overhead and latency as well as improving data privacy. However, deploying distributed learning over wireless networks faces several challenges including the uncertain wireless environment, limited wireless resources (e.g., transmit power and radio spectrum), and hardware resources. This paper provides a comprehensive study of how distributed learning can be efficiently and effectively deployed over wireless edge networks. We present a detailed overview of several emerging distributed learning paradigms, including federated learning, federated distillation, distributed inference, and multi-agent reinforcement learning. For each learning framework, we first introduce the motivation for deploying it over wireless networks. Then, we present a detailed literature review on the use of communication techniques for its efficient deployment. We then introduce an illustrative example to show how to optimize wireless networks to improve its performance. Finally, we introduce future research opportunities. In a nutshell, this paper provides a holistic set of guidelines on how to deploy a broad range of distributed learning frameworks over real-world wireless communication networks.
Fudan University（复旦大学）；Shanghai University（上海大学）；Commonwealth Scientific and Industrial Research Organization (CSIRO)（联邦科学与工业研究组织（CSIRO））；University of Manitoba（曼尼托巴大学）
Distributed machine learning (DML) techniques, such as federated learning, partitioned learning, and distributed reinforcement learning, have been increasingly applied to wireless communications. This is due to improved capabilities of terminal devices, explosively growing data volume, congestion in the radio interfaces, and increasing concern of data privacy. The unique features of wireless systems, such as large scale, geographically dispersed deployment, user mobility, and massive amount of data, give rise to new challenges in the design of DML techniques. There is a clear gap in the existi…
This paper proposes a distributed Reinforcement Learning (RL) based framework that can be used for synthesizing MAC layer wireless protocols in IoT networks with low-complexity wireless transceivers. The proposed framework does not rely on complex hardware capabilities such as carrier sensing and its associated algorithmic complexities that are often not supported in wireless transceivers of low-cost and low-energy IoT devices. In this framework, the access protocols are first formulated as Markov Decision Processes (MDP) and then solved using RL. A distributed and multi-Agent RL framework is …
University of York（约克大学）；PUC-Rio（里约热内卢天主教大学）；Princeton University（普林斯顿大学）
This paper proposes a novel distributed reduced--rank scheme and an adaptive algorithm for distributed estimation in wireless sensor networks. The proposed distributed scheme is based on a transformation that performs dimensionality reduction at each agent of the network followed by a reduced-dimension parameter vector. A distributed reduced-rank joint iterative estimation algorithm is developed, which has the ability to achieve significantly reduced communication overhead and improved performance when compared with existing techniques. Simulation results illustrate the advantages of the propo…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；King’s College London（伦敦国王学院）
Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Ga…
Pukyong National University（釜庆国立大学）；Korea University（高丽大学）；Singapore University of Technology and Design（新加坡科技设计大学）
This paper presents a machine learning strategy that tackles a distributed optimization task in a wireless network with an arbitrary number of randomly interconnected nodes. Individual nodes decide their optimal states with distributed coordination among other nodes through randomly varying backhaul links. This poses a technical challenge in distributed universal optimization policy robust to a random topology of the wireless network, which has not been properly addressed by conventional deep neural networks (DNNs) with rigid structural configurations. We develop a flexible DNN formalism terme…
On Privacy, Security, and Trustworthiness in Distributed Wireless Large AI Models (WLAM)
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Key Laboratory of Information Processing, Communication and Networking of Zhejiang Province (IPCAN)（浙江省信息处理、通信与网络重点实验室）；National Mobile Communications Research Laboratory, Southeast University（东南大学国家移动通信重点实验室）；School of Information and Communication Engineering, Beijing University of Posts and Communications（北京邮电大学信息与通信工程学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Center for 6G Technology, Khalifa University of Science and Technology（哈利法科技大学6G技术中心）
Pukyong National University（釜庆国立大学）；Korea University（高丽大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Kyung Hee University（庆熙大学）
This paper studies a deep learning (DL) framework to solve distributed non-convex constrained optimizations in wireless networks where multiple computing nodes, interconnected via backhaul links, desire to determine an efficient assignment of their states based on local observations. Two different configurations are considered: First, an infinite-capacity backhaul enables nodes to communicate in a lossless way, thereby obtaining the solution by centralized computations. Second, a practical finite-capacity backhaul leads to the deployment of distributed solvers equipped along with quantizers fo…
WDMoE: Wireless Distributed Large Language Models with Mixture of Experts
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒介网络创新中心）；Pengcheng Laboratory（鹏城实验室）；Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）未来网络智能研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）
LASER: Linear Compression in Wireless Distributed Optimization
School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）；Department of Electrical and Computer Engineering, UT Austin（德克萨斯大学奥斯汀分校电气与计算机工程系）
Vertical distributed learning exploits the local features collected by multiple learning workers to form a better global model. However, the exchange of data between the workers and the model aggregator for parameter training incurs a heavy communication burden, especially when the learning system is built upon capacity-constrained wireless networks. In this paper, we propose a novel hierarchical distributed learning framework, where each worker separately learns a low-dimensional embedding of their local observed data. Then, they perform communication efficient distributed max-pooling for eff…
The University of Melbourne（墨尔本大学）；Northrop Grumman Corporation（诺斯罗普·格鲁曼公司）
The analysis of distributed techniques is often focused upon their efficiency, without considering their robustness (or lack thereof). Such a consideration is particularly important when devices or central servers can fail, which can potentially cripple distributed systems. When such failures arise in wireless communications networks, important services that they use/provide (like anomaly detection) can be left inoperable and can result in a cascade of security problems. In this paper, we present a novel method to address these risks by combining both flat- and star-topologies, combining the p…
For a large multi-hop wireless network, nodes are preferable to make distributed and localized link-scheduling decisions with only interactions among a small number of neighbors. However, for a slowly decaying channel and densely populated interferers, a small size neighborhood often results in nontrivial link outages and is thus insufficient for making optimal scheduling decisions. A question arises how to deal with the information outside a neighborhood in distributed link-scheduling. In this work, we develop joint approximation of information and distributed link scheduling. We first apply …
Privacy-sensitive data is stored in autonomous vehicles, smart devices, or sensor nodes that can move around with making opportunistic contact with each other. Federation among such nodes was mainly discussed in the context of federated learning with a centralized mechanism in many works. However, because of multi-vendor issues, those nodes do not want to rely on a specific server operated by a third party for this purpose. In this paper, we propose a wireless ad hoc federated learning (WAFL) -- a fully distributed cooperative machine learning organized by the nodes physically nearby. WAFL can…
