State Key Laboratory of Networking and Switching Technology (BUPT)（网络与交换技术国家重点实验室（北京邮电大学））；Key Lab of High Confidence Software Technologies (Peking University)（高可信软件技术重点实验室（北京大学））
Jinliang Yuan, Mengwei Xu, Yuxin Zhao, Kaigui Bian, Gang Huang, Xuanzhe Liu, Shangguang Wang
To preserve user privacy while enabling mobile intelligence, techniques have been proposed to train deep neural networks on decentralized data. However, training over decentralized data makes the design of neural architecture quite difficult as it already was. Such difficulty is further amplified when designing and deploying different neural architectures for heterogeneous mobile platforms. In this work, we propose an automatic neural architecture search into the decentralized training, as a new DNN training paradigm called Federated Neural Architecture Search, namely federated NAS. To deal with the primary challenge of limited on-client computational and communication resources, we present FedNAS, a highly optimized framework for efficient federated NAS. FedNAS fully exploits the key opportunity of insufficient model candidate re-training during the architecture search process, and incorporates three key optimizations: parallel candidates training on partial clients, early dropping candidates with inferior performance, and dynamic round numbers. Tested on large-scale datasets and typical CNN architectures, FedNAS achieves comparable model accuracy as state-of-the-art NAS algorithm that trains models with centralized data, and also reduces the client cost by up to two orders of magnitude compared to a straightforward design of federated NAS.
Federated learning is a distributed machine learning approach to privacy preservation and two major technical challenges prevent a wider application of federated learning. One is that federated learning raises high demands on communication, since a large number of model parameters must be transmitted between the server and the clients. The other challenge is that training large machine learning models such as deep neural networks in federated learning requires a large amount of computational resources, which may be unrealistic for edge devices such as mobile phones. The problem becomes worse w…
Federated Learning (FL) has been proved to be an effective learning framework when data cannot be centralized due to privacy, communication costs, and regulatory restrictions. When training deep learning models under an FL setting, people employ the predefined model architecture discovered in the centralized environment. However, this predefined architecture may not be the optimal choice because it may not fit data with non-identical and independent distribution (non-IID). Thus, we advocate automating federated learning (AutoFL) to improve model accuracy and reduce the manual design effort. We…
Brandeis University（布兰迪斯大学）；Northeastern University（东北大学）；Beihang University（北京航空航天大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Zhejiang University（浙江大学）；University of Windsor（温莎大学）
Neural networks often encounter various stringent resource constraints while deploying on edge devices. To tackle these problems with less human efforts, automated machine learning becomes popular in finding various neural architectures that fit diverse Artificial Intelligence of Things (AIoT) scenarios. Recently, to prevent the leakage of private information while enable automated machine intelligence, there is an emerging trend to integrate federated learning and neural architecture search (NAS). Although promising as it may seem, the coupling of difficulties from both tenets makes the algor…
Federated learning (FL) has recently gained considerable attention due to its ability to learn on decentralised data while preserving client privacy. However, it also poses additional challenges related to the heterogeneity of the participating devices, both in terms of their computational capabilities and contributed data. Meanwhile, Neural Architecture Search (NAS) has been successfully used with centralised datasets, producing state-of-the-art results in constrained or unconstrained settings. However, such centralised datasets may not be always available for training. Most recent work at th…
University of Southern California（南加州大学）；University of Minnesota（明尼苏达大学）
Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, in this work, we introduce SPIDER, an algori…
Neural Architecture Search (NAS) is a collection of methods to craft the way neural networks are built. We apply this idea to Federated Learning (FL), wherein predefined neural network models are trained on the client/device data. This approach is not optimal as the model developers can't observe the local data, and hence, are unable to build highly accurate and efficient models. NAS is promising for FL which can search for global and personalized models automatically for the non-IID data. Most NAS methods are computationally expensive and require fine-tuning after the search, making it a two-…
University of California San Diego（加州大学圣地亚哥分校）；Rutgers University（罗格斯大学）
Neural architecture search, which aims to automatically search for architectures (e.g., convolution, max pooling) of neural networks that maximize validation performance, has achieved remarkable progress recently. In many application scenarios, several parties would like to collaboratively search for a shared neural architecture by leveraging data from all parties. However, due to privacy concerns, no party wants its data to be seen by other parties. To address this problem, we propose federated neural architecture search (FNAS), where different parties collectively search for a differentiable…
TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessianAI)（黑森人工智能中心（hessianAI））；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）
Deep neural architectures have profound impact on achieved performance in many of today's AI tasks, yet, their design still heavily relies on human prior knowledge and experience. Neural architecture search (NAS) together with hyperparameter optimization (HO) helps to reduce this dependence. However, state of the art NAS and HO rapidly become infeasible with increasing amount of data being stored in a distributed fashion, typically violating data privacy regulations such as GDPR and CCPA. As a remedy, we introduce FEATHERS - $\textbf{FE}$derated $\textbf{A}$rchi$\textbf{T}$ecture and $\textbf{…
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（安全、可靠与信任跨学科研究中心（SnT），卢森堡大学）；Blekinge Institute of Technology（布莱金厄理工大学）；ADSLabs, Umea University（ADSLabs，乌梅亚大学）
PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning
Beijing Univ. of Posts and Telecomm.（北京邮电大学）；Beijing Jiaotong University（北京交通大学）；Beijing Univ. of Aeronautics and Astronautics（北京航空航天大学）；Tsinghua University（清华大学）
Northeastern University（东北大学）；Sun Yat-sen University（中山大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
To prevent the leakage of private information while enabling automated machine intelligence, there is an emerging trend to integrate federated learning and Neural Architecture Search (NAS). Although promising as it may seem, the coupling of difficulties from both two tenets makes the algorithm development quite challenging. In particular, how to efficiently search the optimal neural architecture directly from massive non-iid data of clients in a federated manner remains to be a hard nut to crack. To tackle this challenge, in this paper, by leveraging the advances in proxy-less NAS, we propose …
We propose a novel training recipe for federated learning with heterogeneous networks where each device can have different architectures. We introduce training with a side objective to the devices of higher complexities to jointly train different architectures in a federated setting. We empirically show that our approach improves the performance of different architectures and leads to high communication savings compared to the state-of-the-art methods.
Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；University of New South Wales（新南威尔士大学）
Federated learning is an emerging machine learning paradigm that enables multiple devices to train models locally and formulate a global model, without sharing the clients' local data. A federated learning system can be viewed as a large-scale distributed system, involving different components and stakeholders with diverse requirements and constraints. Hence, developing a federated learning system requires both software system design thinking and machine learning knowledge. Although much effort has been put into federated learning from the machine learning perspectives, our previous systematic…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）
Recently, some Neural Architecture Search (NAS) techniques are proposed for the automatic design of Graph Convolutional Network (GCN) architectures. They bring great convenience to the use of GCN, but could hardly apply to the Federated Learning (FL) scenarios with distributed and private datasets, which limit their applications. Moreover, they need to train many candidate GCN models from scratch, which is inefficient for FL. To address these challenges, we propose FL-AGCNS, an efficient GCN NAS algorithm suitable for FL scenarios. FL-AGCNS designs a federated evolutionary optimization strateg…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Chungnam National University（忠南大学）
Federated learning has emerged as a new paradigm of collaborative machine learning; however, many prior studies have used global aggregation along a star topology without much consideration of the communication scalability or the diurnal property relied on clients' local time variety. In contrast, ring architecture can resolve the scalability issue and even satisfy the diurnal property by iterating nodes without an aggregation. Nevertheless, such ring-based algorithms can inherently suffer from the high-variance problem. To this end, we propose a novel algorithm called TornadoAggregate that im…
AIGC-assisted Federated Learning for Edge Intelligence: Architecture Design, Research Challenges and Future Directions
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Center for Intelligent Networking and Communications (CINC), University of Electronic Science and Technology of China（电子科技大学智能组网与通信中心）
Federated learning is a distributed machine learning method in which a single server and multiple clients collaboratively build machine learning models without sharing datasets on clients. Numerous methods have been proposed to cope with the data heterogeneity issue in federated learning. Existing solutions require a model architecture tuned by the central server, yet a major technical challenge is that it is difficult to tune the model architecture due to the absence of local data on the central server. In this paper, we propose Federated learning via Model exchange (FedMe), which personalize…
A decentralized federated learning architecture is proposed to apply to the Businesses-to-Businesses scenarios by introducing the consortium blockchain in this paper. We introduce a model verification mechanism to ensure the quality of local models trained by participators. To analyze the latency of the system, a latency model is constructed by considering the work flow of the architecture. Finally the experiment results show that our latency model does well in quantifying the actual delays.
National University of Singapore（新加坡国立大学）；ETH Zurich（苏黎世联邦理工学院）；Institute for Infocomm Research（资讯通信研究院）
Federated Reinforcement Learning (FedRL) encourages distributed agents to learn collectively from each other's experience to improve their performance without exchanging their raw trajectories. The existing work on FedRL assumes that all participating agents are homogeneous, which requires all agents to share the same policy parameterization (e.g., network architectures and training configurations). However, in real-world applications, agents are often in disagreement about the architecture and the parameters, possibly also because of disparate computational budgets. Because homogeneity is not…
With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model wi…
Federated learning is an emerging privacy-preserving AI technique where clients (i.e., organisations or devices) train models locally and formulate a global model based on the local model updates without transferring local data externally. However, federated learning systems struggle to achieve trustworthiness and embody responsible AI principles. In particular, federated learning systems face accountability and fairness challenges due to multi-stakeholder involvement and heterogeneity in client data distribution. To enhance the accountability and fairness of federated learning systems, we pre…
LinkedIn has grown to become a platform hosting diverse sources of information ranging from member profiles, jobs, professional groups, slideshows etc. Given the existence of multiple sources, when a member issues a query like "software engineer", the member could look for software engineer profiles, jobs or professional groups. To tackle this problem, we exploit a data-driven approach that extracts searcher intents from their profile data and recent activities at a large scale. The intents such as job seeking, hiring, content consuming are used to construct features to personalize federated s…
The longstanding goals of federated learning (FL) require rigorous privacy guarantees and low communication overhead while holding a relatively high model accuracy. However, simultaneously achieving all the goals is extremely challenging. In this paper, we propose a novel framework called hierarchical federated learning (H-FL) to tackle this challenge. Considering the degradation of the model performance due to the statistic heterogeneity of the training data, we devise a runtime distribution reconstruction strategy, which reallocates the clients appropriately and utilizes mediators to rearran…
Information Sciences Institute（信息科学研究所）；University of Southern California（南加州大学）；Imaging Genetics Center（影像遗传学中心）；USC Stevens Neuroimaging and Informatics Institute（南加州大学史蒂文斯神经影像与信息学研究所）
The amount of biomedical data continues to grow rapidly. However, the ability to analyze these data is limited due to privacy and regulatory concerns. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning is a promising approach to learn a joint model over data silos. This architecture does not share any subject data across sites, only aggregated parameters, often in encrypted environments, thus satisfying privacy and regulatory requirements. Here, we describe our Federated Learning architecture and tr…
University of Electronic Science and Technology of China（电子科技大学）；Pengcheng Laboratory（鹏城实验室）；Mohamed Bin Zayed University Of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Federated learning is gaining popularity as a distributed machine learning method that can be used to deploy AI-dependent IoT applications while protecting client data privacy and security. Due to the differences of clients, a single global model may not perform well on all clients, so the personalized federated learning method, which trains a personalized model for each client that better suits its individual needs, becomes a research hotspot. Most personalized federated learning research, however, focuses on data heterogeneity while ignoring the need for model architecture heterogeneity. Mos…
University of Murcia（穆尔西亚大学）；University of Zurich（苏黎世大学）；armasuisse Science & Technology（瑞士联邦国防采购局科技部门）
Federated learning (FL) allows participants to collaboratively train machine and deep learning models while protecting data privacy. However, the FL paradigm still presents drawbacks affecting its trustworthiness since malicious participants could launch adversarial attacks against the training process. Related work has studied the robustness of horizontal FL scenarios under different attacks. However, there is a lack of work evaluating the robustness of decentralized vertical FL and comparing it with horizontal FL architectures affected by adversarial attacks. Thus, this work proposes three d…
Standard Federated Learning (FL) techniques are limited to clients with identical network architectures. This restricts potential use-cases like cross-platform training or inter-organizational collaboration when both data privacy and architectural proprietary are required. We propose a new FL framework that accommodates heterogeneous client architecture by adopting a graph hypernetwork for parameter sharing. A property of the graph hyper network is that it can adapt to various computational graphs, thereby allowing meaningful parameter sharing across models. Unlike existing solutions, our fram…
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；UC Santa Cruz（加州大学圣克鲁兹分校）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigoro…
Federated Learning Architectures: A Performance Evaluation with Crop Yield Prediction Application
Department of Computer Science Mahishadal Raj College, Mahishadal, West Bengal, 721628, India；Distributed Systems (CLOUDS) Laboratory, \ of Computing；Information Systems, The University of Melbourne, Victoria, 3052, Australia
ConsenSys Health；Imperial College London（帝国理工学院）；UC San Diego（加州大学圣地亚哥分校）；Johns Hopkins Berman Institute of Bioethics（约翰斯·霍普金斯大学伯曼生物伦理研究所）
We propose a novel architecture for federated learning within healthcare consortia. At the heart of the solution is a unique integration of privacy preserving technologies, built upon native enterprise blockchain components available in the Ethereum ecosystem. We show how the specific characteristics and challenges of healthcare consortia informed our design choices, notably the conception of a new Secure Aggregation protocol assembled with a protected hardware component and an encryption toolkit native to Ethereum. Our architecture also brings in a privacy preserving audit trail that logs eve…
Performance and Energy Trade-Off Analysis of Hierarchical Federated Learning for Plant Disease Classification
University of Zagreb（扎格雷布大学）；University of Thessaly（塞萨洛尼基大学）；University of Glasgow（格拉斯哥大学）；Trinity College Dublin（都柏林信任学院）；Italian National Research Council（意大利国家研究委员会）
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Federated learning involves a central processor that works with multiple agents to find a global model. The process consists of repeatedly exchanging estimates, which results in the diffusion of information pertaining to the local private data. Such a scheme can be inconvenient when dealing with sensitive data, and therefore, there is a need for the privatization of the algorithms. Furthermore, the current architecture of a server connected to multiple clients is highly sensitive to communication failures and computational overloads at the server. Thus in this work, we develop a private multi-…
Principles and Components of Federated Learning Architectures
Research and Development Department（研发部）；University of Hertfordshire（赫特福德大学）；Bangladesh University of Engineering and Technology（孟加拉工程与技术大学）；Westcliff university（韦斯特克莱夫大学）；Clark Atlanta University（克拉克亚特兰大大学）
Boston University（波士顿大学）；Saint Louis University（圣路易斯大学）
Everyday, large amounts of sensitive data is distributed across mobile phones, wearable devices, and other sensors. Traditionally, these enormous datasets have been processed on a single system, with complex models being trained to make valuable predictions. Distributed machine learning techniques such as Federated and Split Learning have recently been developed to protect user data and privacy better while ensuring high performance. Both of these distributed learning architectures have advantages and disadvantages. In this paper, we examine these tradeoffs and suggest a new hybrid Federated S…
