Hierarchical federated learning (HFL) enables distributed training of models across multiple devices with the help of several edge servers and a cloud edge server in a privacy-preserving manner. In this paper, we consider HFL with highly mobile devices, mainly targeting at vehicular networks. Through convergence analysis, we show that mobility influences the convergence speed by both fusing the edge data and shuffling the edge models. While mobility is usually considered as a challenge from the perspective of communication, we prove that it increases the convergence speed of HFL with edge-level heterogeneous data, since more diverse data can be incorporated. Furthermore, we demonstrate that a higher speed leads to faster convergence, since it accelerates the fusion of data. Simulation results show that mobility increases the model accuracy of HFL by up to 15.1% when training a convolutional neural network on the CIFAR-10 dataset.
We propose Federated Accelerated Stochastic Gradient Descent (FedAc), a principled acceleration of Federated Averaging (FedAvg, also known as Local SGD) for distributed optimization. FedAc is the first provable acceleration of FedAvg that improves convergence speed and communication efficiency on various types of convex functions. For example, for strongly convex and smooth functions, when using $M$ workers, the previous state-of-the-art FedAvg analysis can achieve a linear speedup in $M$ if given $M$ rounds of synchronization, whereas FedAc only requires $M^{\frac{1}{3}}$ rounds. Moreover, we…
FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
School of Artificial Intelligence Xi'an Xidian University, China；College of Intelligence；Computing Tianjin Tianjin University, China；Medical College, Tianjin University\ Cheng Laboratory Tianjin China；University of Southern California Los Angeles US；School of Artificial Intelligence；Medical College, Tianjin University\ Cheng Laboratory；University of Southern California
Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular. While in other contexts the best performing gradient-type methods invariably rely on some form of acceleration/momentum to reduce the number of iterations, there are no methods which combine the benefits of both gradient compression and acceleration. In this paper, we remedy this situation and propose the first accelerated compressed gradient descent (ACGD) methods. In the single machine regime, we prove that ACGD enjoys t…
Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of heterogeneous devices; here, the need to keep data on device and perform local training makes it difficult to efficiently train and evaluate configurations. In this work, we investigate the problem of federated hyperparameter tuning. We first identify key challenges and show how standard approaches may be adapted to form baselines for the federated setting. Then, by making a nove…
University of Science and Technology of China（中国科学技术大学）；University of Warwick（华威大学）
Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this paper, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence pr…
Duzce University（杜兹切大学）；University of Luxembourg（卢森堡大学）；Imperial College London（帝国理工学院）；University of Oulu（奥卢大学）
Machine learning (ML) has recently been adopted in vehicular networks for applications such as autonomous driving, road safety prediction and vehicular object detection, due to its model-free characteristic, allowing adaptive fast response. However, most of these ML applications employ centralized learning (CL), which brings significant overhead for data transmission between the parameter server and vehicular edge devices. Federated learning (FL) framework has been recently introduced as an efficient tool with the goal of reducing transmission overhead while achieving privacy through the trans…
University of Pittsburgh（匹兹堡大学）；Duke University（杜克大学）
Mobile crowdsensing has gained significant attention in recent years and has become a critical paradigm for emerging Internet of Things applications. The sensing devices continuously generate a significant quantity of data, which provide tremendous opportunities to develop innovative intelligent applications. To utilize these data to train machine learning models while not compromising user privacy, federated learning has become a promising solution. However, there is little understanding of whether federated learning algorithms are guaranteed to converge. We reconsider model averaging in fede…
Adaptive and Parallel Split Federated Learning in Vehicular Edge Computing
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Xidian University（西安电子科技大学）；Guangzhou Institute of Technology, Xidian University（西安电子科技大学广州研究院）；Faculty of Information Technology, University of Jyväskylä（于韦斯屈莱大学信息技术学院）
University of Oulu（奥卢大学）；University of the Basque Country UPV/EHU（巴斯克大学）；Virginia Tech（弗吉尼亚理工大学）
Cooperative perception plays a vital role in extending a vehicle's sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared?, at which resolution?, and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperati…
Jiangnan University（江南大学）；State Key Laboratory of Integrated Services Networks (Xidian University)（综合业务网理论及关键技术国家重点实验室（西安电子科技大学））；Qualcomm（高通公司）；Tsinghua University（清华大学）；University of Kent（肯特大学）；Banma Network Technology Co., Ltd.（斑马网络技术有限公司）
The vehicular edge computing (VEC) can cache contents in different RSUs at the network edge to support the real-time vehicular applications. In VEC, owing to the high-mobility characteristics of vehicles, it is necessary to cache the user data in advance and learn the most popular and interesting contents for vehicular users. Since user data usually contains privacy information, users are reluctant to share their data with others. To solve this problem, traditional federated learning (FL) needs to update the global model synchronously through aggregating all users' local models to protect user…
Semantic Communication-Enhanced Split Federated Learning for Vehicular Networks: Architecture, Challenges, and Case Study
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Faculty of Information Technology, University of Jyväskylä（于韦斯屈莱大学信息科技学院）；Center for Intelligent Networking and Communications (CINC), University of Electronic Science and Technology of China（电子科技大学智能网络与通信中心）
University of Sindh（信德大学）；Munster Technological University（芒斯特理工大学）；Nearby Computing（Nearby Computing公司）
The applications concerning vehicular networks benefit from the vision of beyond 5G and 6G technologies such as ultra-dense network topologies, low latency, and high data rates. Vehicular networks have always faced data privacy preservation concerns, which lead to the advent of distributed learning techniques such as federated learning. Although federated learning has solved data privacy preservation issues to some extent, the technique is quite vulnerable to model inversion and model poisoning attacks. We assume that the design of defense mechanism and attacks are two sides of the same coin. …
Carnegie Mellon University（卡内基梅隆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Washington（华盛顿大学）
We consider strongly convex-concave minimax problems in the federated setting, where the communication constraint is the main bottleneck. When clients are arbitrarily heterogeneous, a simple Minibatch Mirror-prox achieves the best performance. As the clients become more homogeneous, using multiple local gradient updates at the clients significantly improves upon Minibatch Mirror-prox by communicating less frequently. Our goal is to design an algorithm that can harness the benefit of similarity in the clients while recovering the Minibatch Mirror-prox performance under arbitrary heterogeneity (…
Most existing federated learning methods assume that clients have fully labeled data to train on, while in reality, it is hard for the clients to get task-specific labels due to users' privacy concerns, high labeling costs, or lack of expertise. This work considers the server with a small labeled dataset and intends to use unlabeled data in multiple clients for semi-supervised learning. We propose a new framework with a generalized model, Federated Incremental Learning (FedIL), to address the problem of how to utilize labeled data in the server and unlabeled data in clients separately in the s…
Chongqing University of Posts and Telecommunications（重庆邮电大学）
Federated learning (FL), which has gained increasing attention recently, enables distributed devices to train a common machine learning (ML) model for intelligent inference cooperatively without data sharing. However, problems in practical networks, such as non-independent-and-identically-distributed (non-iid) raw data and limited bandwidth, give rise to slow and unstable convergence of the FL training process. To address these issues, we propose a new FL method that can significantly mitigate statistical heterogeneity through the depersonalization mechanism. Particularly, we decouple the glob…
Jiangnan University（江南大学）；State Key Laboratory of Integrated Services Networks (Xidian University)（综合业务网理论及关键技术国家重点实验室（西安电子科技大学））；Qualcomm（高通公司）；Tsinghua University（清华大学）；University of Kent（肯特大学）
Vehicular networks enable vehicles support real-time vehicular applications through training data. Due to the limited computing capability, vehicles usually transmit data to a road side unit (RSU) at the network edge to process data. However, vehicles are usually reluctant to share data with each other due to the privacy issue. For the traditional federated learning (FL), vehicles train the data locally to obtain a local model and then upload the local model to the RSU to update the global model, thus the data privacy can be protected through sharing model parameters instead of data. The tradi…
Optimizing Age of Information in Vehicular Edge Computing with Federated Graph Neural Network Multi-Agent Reinforcement Learning
School of Internet of Things Engineering, Jiangnan University（江南大学物联网工程学院）；Tsinghua University（清华大学）；Xidian University（西安电子科技大学）；Shanghai Jiao Tong University（上海交通大学）；University of Kent（肯特大学）；Hong Kong University of Science and Technology（香港科技大学）
Fast convergence of a Federated Expectation-Maximization Algorithm
Institute of Mathematics, Ecole Polytechnique Federale de Lausanne（洛桑联邦理工学院数学研究所）；Department of Operations Research and Financial Engineering（运筹学与金融工程系）；Princeton University（普林斯顿大学）
Recently, federated learning (FL) has received intensive research because of its ability in preserving data privacy for scattered clients to collaboratively train machine learning models. Commonly, a parameter server (PS) is deployed for aggregating model parameters contributed by different clients. Decentralized federated learning (DFL) is upgraded from FL which allows clients to aggregate model parameters with their neighbours directly. DFL is particularly feasible for vehicular networks as vehicles communicate with each other in a vehicle-to-vehicle (V2V) manner. However, due to the restric…
Noise-Resilient Quantum Aggregation on NISQ for Federated ADAS Learning
Research Scholar, Dept. of CSE（研究学者，计算机科学与工程系）；Dept. of CSE, School of Engineering（计算机科学与工程系，工程学院）；PES University, RR Nagar, Bangalore, India（PES大学）；Dayananda Sagar University, Bangalore, India（Dayananda Sagar大学）
Model Partition and Resource Allocation for Split Learning in Vehicular Edge Networks
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Microelectronics and Communication Engineering, Chongqing University（重庆大学微电子与通信工程学院）；Department of Computer Science, University of Exeter（埃克塞特大学计算机科学系）
Federated learning (FL) is a machine learning technique that aims at training an algorithm across decentralized entities holding their local data private. Wireless mobile networks allow users to communicate with other fixed or mobile users. The road traffic network represents an infrastructure-based configuration of a wireless mobile network where the Connected and Automated Vehicles (CAV) represent the communicating entities. Applying FL in a wireless mobile network setting gives rise to a new threat in the mobile environment that is very different from the traditional fixed networks. The thr…
Dynamic Scheduling for Vehicle-to-Vehicle Communications Enhanced Federated Learning
Beijing National Research Center for Information Science and Technology, Department of Electronic Engineering, Tsinghua University（北京信息科学与技术国家研究中心，电子工程系，清华大学）；School of Electronic and Information Engineering, Beijing Jiaotong University（电子信息工程学院，北京交通大学）
Federated learning (FL) brings collaborative intelligence into industries without centralized training data to accelerate the process of Industry 4.0 on the edge computing level. FL solves the dilemma in which enterprises wish to make the use of data intelligence with security concerns. To accelerate industrial Internet of things with the further leverage of FL, existing achievements on FL are developed from three aspects: 1) define terminologies and elaborate a general framework of FL for accommodating various scenarios; 2) discuss the state-of-the-art of FL on fundamental researches includin…
Auburn University（奥本大学）；Sony AI（索尼人工智能）；Baidu Research（百度研究院）；University of Oregon（俄勒冈大学）
The federated learning (FL) framework enables edge clients to collaboratively learn a shared inference model while keeping privacy of training data on clients. Recently, many heuristics efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings for improving convergence and accuracy. However, there is still a paucity of theoretical principles on where to and how to design and utilize adaptive optimization methods in federated settings. This work aims to develop novel adaptive optimization methods for FL from the pers…
Fraunhofer Institute for Transportation and Infrastructure Systems IVI（弗劳恩霍夫交通与基础设施系统研究所IVI）；Technical University of Munich（慕尼黑工业大学）；Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）
Deep learning is a key approach for the environment perception function of Cooperative Intelligent Transportation Systems (C-ITS) with autonomous vehicles and smart traffic infrastructure. In today's C-ITS, smart traffic participants are capable of timely generating and transmitting a large amount of data. However, these data can not be used for model training directly due to privacy constraints. In this paper, we introduce a federated learning framework coping with Hierarchical Heterogeneity (H2-Fed), which can notably enhance the conventional pre-trained deep learning model. The framework ex…
Accelerated Training of Federated Learning via Second-Order Methods
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, India（人工智能系，印度理工学院海得拉尔分校）；Department of Computing Technologies, Swinburne University of Technology, Hawthorn, australia（计算技术系，斯威本科技大学 Hawthorn 分校）；Department of Mechanical Engineering , Indian Institute of Technology Delhi, India（机械工程系，印度理工学院德里分校）；Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India（计算机科学与工程系，印度理工学院海得拉尔分校）
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Intel Labs（英特尔研究院）；University of Central Florida（中佛罗里达大学）
Federated learning (FL) is a distributed machine learning technology for next-generation AI systems that allows a number of workers, i.e., edge devices, collaboratively learn a shared global model while keeping their data locally to prevent privacy leakage. Enabling FL over wireless multi-hop networks can democratize AI and make it accessible in a cost-effective manner. However, the noisy bandwidth-limited multi-hop wireless connections can lead to delayed and nomadic model updates, which significantly slows down the FL convergence speed. To address such challenges, this paper aims to accelera…
Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；Imaging Genetics Center, Stevens Neuroimaging and Informatics Institute, University of Southern California（南加州大学史蒂文斯神经影像与信息学研究所影像遗传学中心）
There are situations where data relevant to machine learning problems are distributed across multiple locations that cannot share the data due to regulatory, competitiveness, or privacy reasons. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning (FL) is a promising approach to learn a joint model over all the available data across silos. In many cases, the sites participating in a federation have different data distributions and computational capabilities. In these heterogeneous environments, existi…
