Federated learning involves training statistical models over remote devices or siloed data centers, such as mobile phones or hospitals, while keeping data localized. Training in heterogeneous and potentially massive networks introduces novel challenges that require a fundamental departure from standard approaches for large-scale machine learning, distributed optimization, and privacy-preserving data analysis. In this article, we discuss the unique characteristics and challenges of federated learning, provide a broad overview of current approaches, and outline several directions of future work that are relevant to a wide range of research communities.
Pennsylvania State University（宾夕法尼亚州立大学）；Fudan University（复旦大学）
Federated learning (FL) and split learning (SL) are two emerging collaborative learning methods that may greatly facilitate ubiquitous intelligence in Internet of Things (IoT). Federated learning enables machine learning (ML) models locally trained using private data to be aggregated into a global model. Split learning allows different portions of an ML model to be collaboratively trained on different workers in a learning framework. Federated learning and split learning, each has unique advantages and respective limitations, may complement each other toward ubiquitous intelligence in IoT. The…
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Conventional machine learning techniques are conducted in a centralized manner. Recently, the massive volume of generated wireless data, the privacy concerns and the increasing computing capabilities of wireless end-devices have led to the emergence of a promising decentralized solution, termed as Wireless Federated Learning (WFL). In this first of the two parts paper, we present the application of WFL in the sixth generation of wireless networks (6G), which is envisioned to be an integrated communication and computing platform. After analyzing the key concepts of WFL, we discuss the core chal…
Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.
Shell Global Solutions International B.V.（壳牌全球解决方案国际有限公司）；Delft University of Technology (TU Delft)（代尔夫特理工大学）；Myant Inc.（迈安特公司）
Classical federated learning approaches incur significant performance degradation in the presence of non-IID client data. A possible direction to address this issue is forming clusters of clients with roughly IID data. Most solutions following this direction are iterative and relatively slow, also prone to convergence issues in discovering underlying cluster formations. We introduce federated learning with taskonomy (FLT) that generalizes this direction by learning the task-relatedness between clients for more efficient federated aggregation of heterogeneous data. In a one-off process, the ser…
AIGC-assisted Federated Learning for Edge Intelligence: Architecture Design, Research Challenges and Future Directions
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Center for Intelligent Networking and Communications (CINC), University of Electronic Science and Technology of China（电子科技大学智能组网与通信中心）
Australian Department of Health（澳大利亚卫生部）；University of Technology Sydney（悉尼科技大学）
Privacy protection is an ethical issue with broad concern in Artificial Intelligence (AI). Federated learning is a new machine learning paradigm to learn a shared model across users or organisations without direct access to the data. It has great potential to be the next-general AI model training framework that offers privacy protection and therefore has broad implications for the future of digital health and healthcare informatics. Implementing an open innovation framework in the healthcare industry, namely open health, is to enhance innovation and creative capability of health-related organi…
Federated Large Language Models: Current Progress and Future Directions
Carnegie Mellon University（卡内基梅隆大学）；Duke University（杜克大学）；University of California San Diego（加州大学圣地亚哥分校）；The University of New South Wales（新南威尔士大学）；Adobe Research（Adobe研究）；University of Maryland College Park（马里兰大学学院公园分校）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）
Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions
Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学先进技术网络空间研究院）；Huangpu Research School of Guangzhou University（广州大学黄埔研究院）；Iwate Biotechnology Research Center（岩手生物技术研究中心）；Nanjing University of Posts and Telecommunications（南京邮电大学）
As the 5G communication networks are being widely deployed worldwide, both industry and academia have started to move beyond 5G and explore 6G communications. It is generally believed that 6G will be established on ubiquitous Artificial Intelligence (AI) to achieve data-driven Machine Learning (ML) solutions in heterogeneous and massive-scale networks. However, traditional ML techniques require centralized data collection and processing by a central server, which is becoming a bottleneck of large-scale implementation in daily life due to significantly increasing privacy concerns. Federated lea…
Federated learning data is drawn from a distribution of distributions: clients are drawn from a meta-distribution, and their data are drawn from local data distributions. Thus generalization studies in federated learning should separate performance gaps from unseen client data (out-of-sample gap) from performance gaps from unseen client distributions (participation gap). In this work, we propose a framework for disentangling these performance gaps. Using this framework, we observe and explain differences in behavior across natural and synthetic federated datasets, indicating that dataset synth…
We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication. Our approach - drawing on ensemble learning and knowledge aggregation - achieves an average relative gain of 51.5% in AUC over local baselines and comes within 90.1% of the (unattainable) global ideal. We discuss these methods and identify several promising directions of future work.
Generative AI like ChatGPT in Blockchain Federated Learning: use cases, opportunities and future
Southern Illinois University（南伊利诺伊大学）；University of Asia Pacific（亚太大学）；Green University of Bangladesh（孟加拉国绿色大学）；University of North Texas（北得克萨斯大学）；John Hopkins University（约翰霍普金斯大学）
VinUniversity（文大大学）；Hanoi University of Science and Technology（河内科技大学）
Federated learning (FL) is a machine learning (ML) approach that allows the use of distributed data without compromising personal privacy. However, the heterogeneous distribution of data among clients in FL can make it difficult for the orchestration server to validate the integrity of local model updates, making FL vulnerable to various threats, including backdoor attacks. Backdoor attacks involve the insertion of malicious functionality into a targeted model through poisoned updates from malicious clients. These attacks can cause the global model to misbehave on specific inputs while appeari…
Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（中国教育部系统控制与信息处理重点实验室）；Shanghai Engineering Research Center of Intelligent Control and Management（上海智能控制与管理工程技术研究中心）；Shanghai Dianji University（上海电机学院）
Federated learning protects data privacy and security by exchanging models instead of data. However, unbalanced data distributions among participating clients compromise the accuracy and convergence speed of federated learning algorithms. To alleviate this problem, unlike previous studies that limit the distance of updates for local models, we propose global-update-guided federated learning (FedGG), which introduces a model-cosine loss into local objective functions, so that local models can fit local data distributions under the guidance of update directions of global models. Furthermore, con…
Federated learning is an emerging distributed machine learning framework for privacy preservation. However, models trained in federated learning usually have worse performance than those trained in the standard centralized learning mode, especially when the training data are not independent and identically distributed (Non-IID) on the local devices. In this survey, we pro-vide a detailed analysis of the influence of Non-IID data on both parametric and non-parametric machine learning models in both horizontal and vertical federated learning. In addition, cur-rent research work on handling chall…
Communication efficiency is crucial for federated learning (FL). Conducting local training steps in clients to reduce the communication frequency between clients and the server is a common method to address this issue. However, this strategy leads to the client drift problem due to \textit{non-i.i.d.} data distributions in different clients which severely deteriorates the performance. In this work, we propose a new method to improve the training performance in cross-silo FL via maintaining double momentum buffers. In our algorithm, one momentum buffer is used to track the server model updating…
Federated Learning enables training of a general model through edge devices without sending raw data to the cloud. Hence, this approach is attractive for digital health applications, where data is sourced through edge devices and users care about privacy. Here, we report on the feasibility to train deep neural networks on the Raspberry Pi4s as edge devices. A CNN, a LSTM and a MLP were successfully trained on the MNIST data-set. Further, federated learning is demonstrated experimentally on IID and non-IID samples in a parametric study, to benchmark the model convergence. The weight updates fro…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Arizona State University（亚利桑那州立大学）
Directly motivated by security-related applications from the Homeland Security Enterprise, we focus on the privacy-preserving analysis of graph data, which provides the crucial capacity to represent rich attributes and relationships. In particular, we discuss two directions, namely privacy-preserving graph generation and federated graph learning, which can jointly enable the collaboration among multiple parties each possessing private graph data. For each direction, we identify both "quick wins" and "hard problems". Towards the end, we demonstrate a user interface that can facilitate model exp…
Towards One-shot Federated Learning: Advances, Challenges, and Future Directions
Department of Electrical Engineering and Information Technology, University of Naples Federico II（那不勒斯费德里科二世大学电气工程与信息科技系）；Department of Mathematics and Applications "R. Caccioppoli", University of Naples Federico II（那不勒斯费德里科二世大学数学与应用系）；Department of Mathematics, Indian Institute of Technology Indore（印度理工学院Indore分校数学系）
The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning
University of Michigan（密歇根大学）；University of Wisconsin Madison（威斯康星大学麦迪逊分校）；American University of Beirut（贝鲁特美国大学）；National University of Singapore（新加坡国立大学）
University of Bristol（布里斯托大学）；Toshiba Europe Ltd.（东芝欧洲有限公司）
Data-enabled cities are recently accelerated and enhanced with automated learning for improved Smart Cities applications. In the context of an Internet of Things (IoT) ecosystem, the data communication is frequently costly, inefficient, not scalable and lacks security. Federated Learning (FL) plays a pivotal role in providing privacy-preserving and communication efficient Machine Learning (ML) frameworks. In this paper we evaluate the feasibility of FL in the context of a Smart Cities Street Light Monitoring application. FL is evaluated against benchmarks of centralised and (fully) personalise…
National University of Singapore（新加坡国立大学）；The University of Western Australia（西澳大利亚大学）
Federated learning has been a hot research topic in enabling the collaborative training of machine learning models among different organizations under the privacy restrictions. As researchers try to support more machine learning models with different privacy-preserving approaches, there is a requirement in developing systems and infrastructures to ease the development of various federated learning algorithms. Similar to deep learning systems such as PyTorch and TensorFlow that boost the development of deep learning, federated learning systems (FLSs) are equivalently important, and face challen…
Harbin Institute of Technology（哈尔滨工业大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
As an emerging technology, federated learning (FL) involves training machine learning models over distributed edge devices, which attracts sustained attention and has been extensively studied. However, the heterogeneity of client data severely degrades the performance of FL compared with that in centralized training. It causes the locally trained models of clients to move in different directions. On the one hand, it slows down or even stalls the global updates, leading to inefficient communication. On the other hand, it enlarges the distances between local models, resulting in an aggregated gl…
Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning
University of Helsinki（赫尔辛基大学）；University of Technology Sydney（悉尼科技大学）；Beihang University（北京航空航天大学）；Monash University（莫纳什大学）；Aalto University（阿尔托大学）；Griffith University（格里菲斯大学）；Amazon（亚马逊公司）；Columbia University（哥伦比亚大学）
This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive eval…
Australian Artificial Intelligence Institute (AAII)（澳大利亚人工智能研究所（AAII））
Open banking enables individual customers to own their banking data, which provides fundamental support for the boosting of a new ecosystem of data marketplaces and financial services. In the near future, it is foreseeable to have decentralized data ownership in the finance sector using federated learning. This is a just-in-time technology that can learn intelligent models in a decentralized training manner. The most attractive aspect of federated learning is its ability to decompose model training into a centralized server and distributed nodes without collecting private data. This kind of de…
Carnegie Mellon University（卡内基梅隆大学）；Bosch Center for AI（博世人工智能中心）；Google Research（谷歌研究院）；University of Southern California（南加州大学）；Determined AI
Federated learning aims to jointly learn statistical models over massively distributed remote devices. In this work, we propose FedDANE, an optimization method that we adapt from DANE, a method for classical distributed optimization, to handle the practical constraints of federated learning. We provide convergence guarantees for this method when learning over both convex and non-convex functions. Despite encouraging theoretical results, we find that the method has underwhelming performance empirically. In particular, through empirical simulations on both synthetic and real-world datasets, FedD…
This paper introduces a Modified User Datagram Protocol (UDP) for Federated Learning to ensure efficiency and reliability in the model parameter transport process, maximizing the potential of the Global model in each Federated Learning round. In developing and testing this protocol, the NS3 simulator is utilized to simulate the packet transport over the network and Google TensorFlow is used to create a custom Federated learning environment. In this preliminary implementation, the simulation contains three nodes where two nodes are client nodes, and one is a server node. The results obtained in…
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）；Cincinnati Children's Hospital Medical Center（辛辛那提儿童医院医疗中心）；University of Cincinnati College of Medicine（辛辛那提大学医学院）；University of Cincinnati（辛辛那提大学）
Federated learning (FL) is a system in which a central aggregator coordinates the efforts of multiple clients to solve machine learning problems. This setting allows training data to be dispersed in order to protect privacy. The purpose of this paper is to provide an overview of FL systems with a focus on healthcare. FL is evaluated here based on its frameworks, architectures, and applications. It is shown here that FL solves the preceding issues with a shared global deep learning (DL) model via a central aggregator server. This paper examines recent developments and provides a comprehensive l…
Federated learning has burgeoned recently in machine learning, giving rise to a variety of research topics. Popular optimization algorithms are based on the frameworks of the (stochastic) gradient descent methods or the alternating direction method of multipliers. In this paper, we deploy an exact penalty method to deal with federated learning and propose an algorithm, FedEPM, that enables to tackle four critical issues in federated learning: communication efficiency, computational complexity, stragglers' effect, and data privacy. Moreover, it is proven to be convergent and testified to have h…
Deakin University（迪肯大学）；CSIRO（联邦科学与工业研究组织）；University of New South Wales（新南威尔士大学）；Nanjing University of Science and Technology（南京理工大学）；Nanyang Technological University（南洋理工大学）；Princeton University（普林斯顿大学）
The Industrial Internet of Things (IIoT) offers promising opportunities to transform the operation of industrial systems and becomes a key enabler for future industries. Recently, artificial intelligence (AI) has been widely utilized for realizing intelligent IIoT applications where AI techniques require centralized data collection and processing. However, this is not always feasible in realistic scenarios due to the high scalability of modern IIoT networks and growing industrial data confidentiality. Federated Learning (FL), as an emerging collaborative AI approach, is particularly attractive…
Federated learning (FL) has been developed as a promising framework to leverage the resources of edge devices, enhance customers' privacy, comply with regulations, and reduce development costs. Although many methods and applications have been developed for FL, several critical challenges for practical FL systems remain unaddressed. This paper provides an outlook on FL development, categorized into five emerging directions of FL, namely algorithm foundation, personalization, hardware and security constraints, lifelong learning, and nonstandard data. Our unique perspectives are backed by practic…
From Federated Learning to Quantum Federated Learning for Space-Air-Ground Integrated Networks
Hung Yen University of Technology and Education（兴安技术与教育大学）；University of Tasmania（塔斯马尼亚大学）；University of Alabama in Huntsville（阿拉巴马大学亨茨维尔分校）；Lac Hong University（乐鸿大学）；Manchester Metropolitan University（曼彻斯特城市大学）；International Hellenic University（国际希腊大学）
Purdue University（普渡大学）；NC State University（北卡罗来纳州立大学）
Machine learning (ML) tasks are becoming ubiquitous in today's network applications. Federated learning has emerged recently as a technique for training ML models at the network edge by leveraging processing capabilities across the nodes that collect the data. There are several challenges with employing conventional federated learning in contemporary networks, due to the significant heterogeneity in compute and communication capabilities that exist across devices. To address this, we advocate a new learning paradigm called fog learning which will intelligently distribute ML model training acro…
Federated Learning with New Knowledge: Fundamentals, Advances, and Futures
Northwestern University（西北大学）；Nanyang Technological University（南洋理工大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Hunan University（湖南大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）
NVIDIA（英伟达）；Technical University of Munich (TUM)（慕尼黑工业大学）；ETH Zürich（苏黎世联邦理工学院）；University of Pennsylvania (UPenn)（宾夕法尼亚大学）；Owkin；Vanderbilt University Medical Center（范德比尔特大学医学中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；King’s College London (KCL)（伦敦国王学院）；Intel Corporation（英特尔公司）；Clinical Center, National Institutes of Health (NIH)（美国国立卫生研究院临床中心）；OpenMined；University of Oxford（牛津大学）
Data-driven Machine Learning has emerged as a promising approach for building accurate and robust statistical models from medical data, which is collected in huge volumes by modern healthcare systems. Existing medical data is not fully exploited by ML primarily because it sits in data silos and privacy concerns restrict access to this data. However, without access to sufficient data, ML will be prevented from reaching its full potential and, ultimately, from making the transition from research to clinical practice. This paper considers key factors contributing to this issue, explores how Feder…
