Privacy and bandwidth constraints have led to the use of federated learning (FL) in wireless systems, where training a machine learning (ML) model is accomplished collaboratively without sharing raw data. While using bandwidth-constrained uplink wireless channels, over-the-air (OTA) FL is preferred since the clients can transmit parameter updates simultaneously to a server. A powerful server may not be available for parameter aggregation due to increased latency and server failures. In the absence of a powerful server, decentralised strategy is employed where clients communicate with their neighbors to obtain a consensus ML model while incurring huge communication cost. In this work, we propose the OTA semi-decentralised clustered wireless FL (CWFL) and CWFL-Prox algorithms, which is communication efficient as compared to the decentralised FL strategy, while the parameter updates converge to global minima as O(1/T) for each cluster. Using the MNIST and CIFAR10 datasets, we demonstrate the accuracy performance of CWFL is comparable to the central-server based COTAF and proximal constraint based methods, while beating single-client based ML model by vast margins in accuracy.
Traditionally, clustered federated learning groups clients with the same data distribution into a cluster, so that every client is uniquely associated with one data distribution and helps train a model for this distribution. We relax this hard association assumption to soft clustered federated learning, which allows every local dataset to follow a mixture of multiple source distributions. We propose FedSoft, which trains both locally personalized models and high-quality cluster models in this setting. FedSoft limits client workload by using proximal updates to require the completion of only on…
Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；University of Washington, Seattle（西雅图华盛顿大学）；University of Maryland, College Park（马里兰大学帕克分校）
Knowledge sharing and model personalization are essential components to tackle the non-IID challenge in federated learning (FL). Most existing FL methods focus on two extremes: 1) to learn a shared model to serve all clients with non-IID data, and 2) to learn personalized models for each client, namely personalized FL. There is a trade-off solution, namely clustered FL or cluster-wise personalized FL, which aims to cluster similar clients into one cluster, and then learn a shared model for all clients within a cluster. This paper is to revisit the research of clustered FL by formulating them i…
Personalized federated learning (FL) aims to train model(s) that can perform well for individual clients that are highly data and system heterogeneous. Most work in personalized FL, however, assumes using the same model architecture at all clients and increases the communication cost by sending/receiving models. This may not be feasible for realistic scenarios of FL. In practice, clients have highly heterogeneous system-capabilities and limited communication resources. In our work, we propose a personalized FL framework, PerFed-CKT, where clients can use heterogeneous model architectures and d…
Fedcompass: Federated Clustered and Periodic Aggregation Framework for Hybrid Classical-Quantum Models
1 School of Artificial Intelligence；Computer Science, Jiangnan University 2 Antai College of Economics \& Management, Shanghai Jiao Tong University 3 School of Computer Science, Shanghai Jiao Tong University 4 Cloud Computing；Distributed Systems Laboratory, University of Melbourne
Single-Round Clustered Federated Learning via Data Collaboration Analysis for Non-IID Data
Graduate School of Science and Technology, University of Tsukuba（茨川大学理学技术研究生院）；Center for Artificial Intelligence Research, Tsukuba Institute for Advanced Research, University of Tsukuba（茨川大学先进研究所人工智能研究中心）；Institute of Systems and Information Engineering, University of Tsukuba（茨川大学系统与信息工程研究所）
Novel clustered federated learning based on local loss
LMIB, School of Mathematical Sciences, Beihang University（力学系，数学科学学院，北航）；College of Science, China Agricultural University（中国农业大学，理学院）；School of Mathematical Sciences, Nanjing Normal University（南京师范大学，数学科学学院）；School of Mathematical Sciences, Beihang University（力学系，数学科学学院，北航）
Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)（深圳市人工智能与机器人研究院）；School of Engineering, Westlake University（西湖大学工学院）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）
Institute of Communication Acoustics, Ruhr-Universit\"at Bochum 44780 Bochum, Germany
In this paper we introduce a realistic and challenging, multi-source and multi-room acoustic environment and an improved algorithm for the estimation of source-dominated microphone clusters in acoustic sensor networks. Our proposed clustering method is based on a single microphone per node and on unsupervised clustered federated learning which employs a light-weight autoencoder model. We present an improved clustering control strategy that takes into account the variability of the acoustic scene and allows the estimation of a dynamic range of clusters using reduced amounts of training data. Th…
Texas A&M University（德克萨斯农工大学）；Texas A&M University at Qatar（德克萨斯农工大学卡塔尔分校）；Qatar Foundation Research, Development and Innovation RDI（卡塔尔基金会研究、开发与创新（RDI））；Tennessee Tech University（田纳西理工大学）
This paper proposes using communication pipelining to enhance the wireless spectrum utilization efficiency and convergence speed of federated learning in mobile edge computing applications. Due to limited wireless sub-channels, a subset of the total clients is scheduled in each iteration of federated learning algorithms. On the other hand, the scheduled clients wait for the slowest client to finish its computation. We propose to first cluster the clients based on the time they need per iteration to compute the local gradients of the federated learning model. Then, we schedule a mixture of clie…
This work addresses the problem of optimizing communications between server and clients in federated learning (FL). Current sampling approaches in FL are either biased, or non optimal in terms of server-clients communications and training stability. To overcome this issue, we introduce \textit{clustered sampling} for clients selection. We prove that clustered sampling leads to better clients representatitivity and to reduced variance of the clients stochastic aggregation weights in FL. Compatibly with our theory, we provide two different clustering approaches enabling clients aggregation based…
Distilling A Universal Expert from Clustered Federated Learning
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education（教育部符号计算与知识工程重点实验室）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney（悉尼大学技术学院澳大利亚人工智能研究所）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机科学学院）
University of Padova（帕多瓦大学）；Politecnico di Torino（都灵理工大学）
Federated Learning (FL) has recently emerged as a possible way to tackle the domain shift in real-world Semantic Segmentation (SS) without compromising the private nature of the collected data. However, most of the existing works on FL unrealistically assume labeled data in the remote clients. Here we propose a novel task (FFREEDA) in which the clients' data is unlabeled and the server accesses a source labeled dataset for pre-training only. To solve FFREEDA, we propose LADD, which leverages the knowledge of the pre-trained model by employing self-supervision with ad-hoc regularization techniq…
This paper investigates the robustness of over-the-air federated learning to Byzantine attacks. The simple averaging of the model updates via over-the-air computation makes the learning task vulnerable to random or intended modifications of the local model updates of some malicious clients. We propose a robust transmission and aggregation framework to such attacks while preserving the benefits of over-the-air computation for federated learning. For the proposed robust federated learning, the participating clients are randomly divided into groups and a transmission time slot is allocated to eac…
We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model pa…
A Joint Gradient and Loss Based Clustered Federated Learning Design
University of Miami（迈阿密大学）；Frost Institute for Data Science and Computing, University of Miami（迈阿密大学弗罗斯特数据科学与计算研究所）；Zhejiang University（浙江大学）；North Carolina State University（北卡罗来纳州立大学）
FedSCAM (Federated Sharpness-Aware Minimization with Clustered Aggregation and Modulation): Scam-resistant SAM for Robust Federated Optimization in Heterogeneous Environments
Department of Computer Science, Lahore University of Management Sciences, Lahore, Pakistan（计算机科学系，拉合尔管理科学大学，拉合尔，巴基斯坦）
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Korea University（高丽大学）
Federated Learning (FL) is currently the most widely adopted framework for collaborative training of (deep) machine learning models under privacy constraints. Albeit it's popularity, it has been observed that Federated Learning yields suboptimal results if the local clients' data distributions diverge. To address this issue, we present Clustered Federated Learning (CFL), a novel Federated Multi-Task Learning (FMTL) framework, which exploits geometric properties of the FL loss surface, to group the client population into clusters with jointly trainable data distributions. In contrast to existin…
Southeast University（东南大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Oulu（奥卢大学）
In this paper, the content popularity prediction problem in fog radio access networks (F-RANs) is investigated. Based on clustered federated learning, we propose a novel mobility-aware popularity prediction policy, which integrates content popularities in terms of local users and mobile users. For local users, the content popularity is predicted by learning the hidden representations of local users and contents. Initial features of local users and contents are generated by incorporating neighbor information with self information. Then, dual-channel neural network (DCNN) model is introduced to …
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）；Baidu Research, Baidu Inc.（百度研究院，百度公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；BOSS Zhipin（BOSS直聘）
Classical federated learning approaches yield significant performance degradation in the presence of Non-IID data distributions of participants. When the distribution of each local dataset is highly different from the global one, the local objective of each client will be inconsistent with the global optima which incur a drift in the local updates. This phenomenon highly impacts the performance of clients. This is while the primary incentive for clients to participate in federated learning is to obtain better personalized models. To address the above-mentioned issue, we present a new algorithm…
This paper considers over-the-air federated learning (OTA-FL). OTA-FL exploits the superposition property of the wireless medium, and performs model aggregation over the air for free. Thus, it can greatly reduce the communication cost incurred in communicating model updates from the edge devices. In order to fully utilize this advantage while providing comparable learning performance to conventional federated learning that presumes model aggregation via noiseless channels, we consider the joint design of transmission scaling and the number of local iterations at each round, given the power con…
UC San Diego（加州大学圣地亚哥分校）；Czech Technical University（捷克技术大学）；UCF（中佛罗里达大学）
Clustered federated learning (FL) has been shown to produce promising results by grouping clients into clusters. This is especially effective in scenarios where separate groups of clients have significant differences in the distributions of their local data. Existing clustered FL algorithms are essentially trying to group together clients with similar distributions so that clients in the same cluster can leverage each other's data to better perform federated learning. However, prior clustered FL algorithms attempt to learn these distribution similarities indirectly during training, which can b…
To reduce negative environmental impacts, power stations and energy grids need to optimize the resources required for power production. Thus, predicting the energy consumption of clients is becoming an important part of every energy management system. Energy usage information collected by the clients' smart homes can be used to train a deep neural network to predict the future energy demand. Collecting data from a large number of distributed clients for centralized model training is expensive in terms of communication resources. To take advantage of distributed data in edge systems, centralize…
Shenzhen University（深圳大学）；Macquarie University（麦考瑞大学）；Georgia State University（佐治亚州立大学）
Federated Learning (FL) is an emerging decentralized learning framework through which multiple clients can collaboratively train a learning model. However, a major obstacle that impedes the wide deployment of FL lies in massive communication traffic. To train high dimensional machine learning models (such as CNN models), heavy communication traffic can be incurred by exchanging model updates via the Internet between clients and the parameter server (PS), implying that the network resource can be easily exhausted. Compressing model updates is an effective way to reduce the traffic amount. Howev…
Edge Intelligence Over the Air: Two Faces of Interference in Federated Learning
Singapore University of Technology and Design（新加坡科技设计大学）；Kyung Hee University（庆熙大学）；Zhejiang University/University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）；Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Singapore University of Technology and Design（新加坡科技设计大学）；ZJU-UIUC Institute, Zhejiang University（浙江大学浙江大学-伊利诺伊大学厄巴纳香槟校区联合学院）
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. But the training efficiency is often throttled by challenges arising from limited communication and data heterogeneity. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck. Additionally, we leverage a bi-level optimization framework to personalize the…
DAG-ACFL: Asynchronous Clustered Federated Learning based on DAG-DLT
Information Countermeasure Technique Institute（信息对抗技术研究所）；School of Cyberspace Science（网络空间科学学院）；Faculty of Computing（计算学部）；Harbin Institute of Technology（哈尔滨工业大学）
