Federated Learning (FL) is an emerging approach for collaboratively training Deep Neural Networks (DNNs) on mobile devices, without private user data leaving the devices. Previous works have shown that non-Independent and Identically Distributed (non-IID) user data harms the convergence speed of the FL algorithms. Furthermore, most existing work on FL measures global-model accuracy, but in many cases, such as user content-recommendation, improving individual User model Accuracy (UA) is the real objective. To address these issues, we propose a Multi-Task FL (MTFL) algorithm that introduces non-federated Batch-Normalization (BN) layers into the federated DNN. MTFL benefits UA and convergence speed by allowing users to train models personalised to their own data. MTFL is compatible with popular iterative FL optimisation algorithms such as Federated Averaging (FedAvg), and we show empirically that a distributed form of Adam optimisation (FedAvg-Adam) benefits convergence speed even further when used as the optimisation strategy within MTFL. Experiments using MNIST and CIFAR10 demonstrate that MTFL is able to significantly reduce the number of rounds required to reach a target UA, by up to $5\times$ when using existing FL optimisation strategies, and with a further $3\times$ improvement when using FedAvg-Adam. We compare MTFL to competing personalised FL algorithms, showing that it is able to achieve the best UA for MNIST and CIFAR10 in all considered scenarios. Finally, we evaluate MTFL with FedAvg-Adam on an edge-computing testbed, showing that its convergence and UA benefits outweigh its overhead.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Criteo AI Lab（Criteo人工智能实验室）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；Ecole Polytechnique（巴黎综合理工学院）
Personalised federated learning (FL) aims at collaboratively learning a machine learning model taylored for each client. Albeit promising advances have been made in this direction, most of existing approaches works do not allow for uncertainty quantification which is crucial in many applications. In addition, personalisation in the cross-device setting still involves important issues, especially for new clients or those having small number of observations. This paper aims at filling these gaps. To this end, we propose a novel methodology coined FedPop by recasting personalised FL into the popu…
Most personalised federated learning (FL) approaches assume that raw data of all clients are defined in a common subspace i.e. all clients store their data according to the same schema. For real-world applications, this assumption is restrictive as clients, having their own systems to collect and then store data, may use heterogeneous data representations. We aim at filling this gap. To this end, we propose a general framework coined FLIC that maps client's data onto a common feature space via local embedding functions. The common feature space is learnt in a federated manner using Wasserstein…
Robust Federated Personalised Mean Estimation for the Gaussian Mixture Model
School of Technology \& Comp. Sc. Tata Institute of Fundamental Research Mumbai, India；Department of Electrical Engineering University of California Los Angeles, USA
Federated learning (FL) is a distributed machine learning approach involving multiple clients collaboratively training a shared model. Such a system has the advantage of more training data from multiple clients, but data can be non-identically and independently distributed (non-i.i.d.). Privacy and integrity preserving features such as differential privacy (DP) and robust aggregation (RA) are commonly used in FL. In this work, we show that on common deep learning tasks, the performance of FL models differs amongst clients and situations, and FL models can sometimes perform worse than local mod…
PTOPOFL: Privacy-Preserving Personalised Federated Learning via Persistent Homology
Université Sorbonne Paris Nord（索邦巴黎北大学）；LAGA（拉加数学研究所）；CNRS（国家科学研究中心）；UMR 7539（7539号研究单位）；Laboratoire d’excellence Infibrex（卓越实验室Infibrex）；Instituto de Hortofruticultura Subtropical y Mediterránea La Mayora (IHSM)（亚得里亚地中海亚热带园艺研究所（IHSM））；Universidad de Málaga–Consejo Superior de Investigaciones Científicas（马拉加大学–高级科学研究理事会）
Federated Learning (FL) has evolved as a promising technique to handle distributed machine learning across edge devices. A single neural network (NN) that optimises a global objective is generally learned in most work in FL, which could be suboptimal for edge devices. Although works finding a NN personalised for edge device specific tasks exist, they lack generalisation and/or convergence guarantees. In this paper, a novel communication efficient FL algorithm for personalised learning in a wireless setting with guarantees is presented. The algorithm relies on finding a ``better`` empirical est…
With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model wi…
University of Bristol（布里斯托大学）；Toshiba Europe Ltd.（东芝欧洲有限公司）
Data-enabled cities are recently accelerated and enhanced with automated learning for improved Smart Cities applications. In the context of an Internet of Things (IoT) ecosystem, the data communication is frequently costly, inefficient, not scalable and lacks security. Federated Learning (FL) plays a pivotal role in providing privacy-preserving and communication efficient Machine Learning (ML) frameworks. In this paper we evaluate the feasibility of FL in the context of a Smart Cities Street Light Monitoring application. FL is evaluated against benchmarks of centralised and (fully) personalise…
Embedding-Based Federated Learning with Runtime Governance for Iron Deficiency Prediction
Theoretical Physics University of Cambridge Cambridge, UK；Translational AI Laboratory, Dept. of Laboratory Medicine Amsterdam UMC Amsterdam, The Netherlands；Precision Health University Research Institute Queen Mary Univ. of London London, UK；Department of Medicine University of Cambridge Cambridge Biomedical Campus Cambridge, UK；Transplant Cambridge Biomedical Campus Cambridge, UK；Dept. of Cardiology Amsterdam Cardiovascular Sciences Amsterdam UMC Amsterdam, The Netherlands
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have receiv…
Federated learning is a method of training a global model from decentralized data distributed across client devices. Here, model parameters are computed locally by each client device and exchanged with a central server, which aggregates the local models for a global view, without requiring sharing of training data. The convergence performance of federated learning is severely impacted in heterogeneous computing platforms such as those at the wireless edge, where straggling computations and communication links can significantly limit timely model parameter updates. This paper develops a novel c…
Chalmers University of Technology（查尔姆斯理工大学）；Zhejiang University（浙江大学）；Research Institutes of Sweden（瑞典研究院）
Bandwidth slicing is introduced to support federated learning in edge computing to assure low communication delay for training traffic. Results reveal that bandwidth slicing significantly improves training efficiency while achieving good learning accuracy.
University of Southern California（南加州大学）；Intel Corporation（英特尔公司）；Broadcom Corporation（博通公司）；Intel Labs（英特尔实验室）；Qualcomm（高通公司）
Federated learning enables training a global model from data located at the client nodes, without data sharing and moving client data to a centralized server. Performance of federated learning in a multi-access edge computing (MEC) network suffers from slow convergence due to heterogeneity and stochastic fluctuations in compute power and communication link qualities across clients. We propose a novel coded computing framework, CodedFedL, that injects structured coding redundancy into federated learning for mitigating stragglers and speeding up the training procedure. CodedFedL enables coded co…
University of Southern California（南加州大学）；Intel Labs（英特尔实验室）
Federated Learning (FL) is an exciting new paradigm that enables training a global model from data generated locally at the client nodes, without moving client data to a centralized server. Performance of FL in a multi-access edge computing (MEC) network suffers from slow convergence due to heterogeneity and stochastic fluctuations in compute power and communication link qualities across clients. A recent work, Coded Federated Learning (CFL), proposes to mitigate stragglers and speed up training for linear regression tasks by assigning redundant computations at the MEC server. Coding redundanc…
FedFusion: Federated Learning with Diversity- and Cluster-Aware Encoders for Robust Adaptation under Label Scarcity
Department of Computer Science University of the Western Cape（计算机科学系乌得勒支大学）；Department of Computer Science Missouri University of Science and Technology（计算机科学系密苏里科技大学）
Pennsylvania State University（宾夕法尼亚州立大学）；Fudan University（复旦大学）
Federated learning (FL) and split learning (SL) are two emerging collaborative learning methods that may greatly facilitate ubiquitous intelligence in Internet of Things (IoT). Federated learning enables machine learning (ML) models locally trained using private data to be aggregated into a global model. Split learning allows different portions of an ML model to be collaboratively trained on different workers in a learning framework. Federated learning and split learning, each has unique advantages and respective limitations, may complement each other toward ubiquitous intelligence in IoT. The…
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Cyberspace Security Research Center, Peng Cheng Laboratory（鹏城实验室网络空间安全研究中心）；School of Electrical and Data Engineering, University of Technology Sydney（悉尼科技大学电气与数据工程学院）；Department of Computer Science and Technology, Fuzhou University（福州大学计算机科学与技术学院）；Department of Computer Science, Electrical and Space Engineering, Lulea University of Technology（吕勒奥理工大学计算机、电气与空间工程系）
In recent years, mobile clients' computing ability and storage capacity have greatly improved, efficiently dealing with some applications locally. Federated learning is a promising distributed machine learning solution that uses local computing and local data to train the Artificial Intelligence (AI) model. Combining local computing and federated learning can train a powerful AI model under the premise of ensuring local data privacy while making full use of mobile clients' resources. However, the heterogeneity of local data, that is, Non-independent and identical distribution (Non-IID) and imb…
Owing to the large volume of sensed data from the enormous number of IoT devices in operation today, centralized machine learning algorithms operating on such data incur an unbearable training time, and thus cannot satisfy the requirements of delay-sensitive inference applications. By provisioning computing resources at the network edge, Mobile Edge Computing (MEC) has become a promising technology capable of collaborating with distributed IoT devices to facilitate federated learning, and thus realize real-time training. However, considering the large volume of sensed data and the limited reso…
HPC-AI Technology Inc.（HPC-AI技术公司）；National University of Singapore（新加坡国立大学）
Federated learning is proposed by Google to safeguard data privacy through training models locally on users' devices. However, with deep learning models growing in size to achieve better results, it becomes increasingly difficult to accommodate the whole model on one single device. Thus, model parallelism is then used to divide the model weights among several devices. With this logic, the approach currently used evenly allocates weights among devices. However, in reality, a computation bottleneck may occur resulting from variant computing power of different users' devices. To address this prob…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Macquarie University（麦考瑞大学）
Pervasive computing applications commonly involve user's personal smartphones collecting data to influence application behavior. Applications are often backed by models that learn from the user's experiences to provide personalized and responsive behavior. While models are often pre-trained on massive datasets, federated learning has gained attention for its ability to train globally shared models on users' private data without requiring the users to share their data directly. However, federated learning requires devices to collaborate via a central server, under the assumption that all users …
FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices using a Computing Power Aware Scheduler
Argonne National Laboratory（阿贡国家实验室）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；VMWare Research（威睿研究院）；The University of Chicago（芝加哥大学）
As a distributed learning approach, federated learning trains a shared learning model over distributed datasets while preserving the training data privacy. We extend the application of federated learning to parking management and introduce FedParking in which Parking Lot Operators (PLOs) collaborate to train a long short-term memory model for parking space estimation without exchanging the raw data. Furthermore, we investigate the management of Parked Vehicle assisted Edge Computing (PVEC) by FedParking. In PVEC, different PLOs recruit PVs as edge computing nodes for offloading services throug…
Decentralized stochastic gradient descent (SGD) is a driving engine for decentralized federated learning (DFL). The performance of decentralized SGD is jointly influenced by inter-node communications and local updates. In this paper, we propose a general DFL framework, which implements both multiple local updates and multiple inter-node communications periodically, to strike a balance between communication efficiency and model consensus. It can provide a general decentralized SGD analytical framework. We establish strong convergence guarantees for the proposed DFL algorithm without the assumpt…
Royal Institute of Technology (KTH)（皇家理工学院（KTH））；ABB Corporate Research Sweden（瑞典ABB企业研究中心）
Driven by the ever-increasing penetration and proliferation of data-driven applications, a new generation of wireless communication, the sixth-generation (6G) mobile system enhanced by artificial intelligence (AI), has attracted substantial research interests. Among various candidate technologies of 6G, low earth orbit (LEO) satellites have appealing characteristics of ubiquitous wireless access. However, the costs of satellite communication (SatCom) are still high, relative to counterparts of ground mobile networks. To support massively interconnected devices with intelligent adaptive learnin…
Blockchain-Enabled Privacy-Preserving Second-Order Federated Edge Learning in Personalized Healthcare
Shanghai Key Laboratory of Intelligent Information Processing Lab, Fudan University, China（上海智能信息处理实验室，复旦大学，中国）；Turku Intelligent Embedded and Robotic Systems (TIERS) Lab, University of Turku, Finland（图尔库智能嵌入式与机器人系统（TIERS）实验室，图尔库大学，芬兰）；TIERS Lab, University of Turku, Finland（TIERS实验室，图尔库大学，芬兰）；Department of Electrical Engineering, Islamic University of Madinah, Saudi Arabia（电气工程系，麦加伊斯兰大学，沙特阿拉伯）；School of Information Science and Technology, Fudan University, Shanghai, China（信息科学与技术学院，复旦大学，上海，中国）；Faculty of Technology, University of Turku, Finland（技术学院，图尔库大学，芬兰）
Since 2014, the NIH funded iDASH (integrating Data for Analysis, Anonymization, SHaring) National Center for Biomedical Computing has hosted yearly competitions on the topic of private computing for genomic data. For one track of the 2020 iteration of this competition, participants were challenged to produce an approach to federated learning (FL) training of genomic cancer prediction models using differential privacy (DP), with submissions ranked according to held-out test accuracy for a given set of DP budgets. More precisely, in this track, we are tasked with training a supervised model for …
Cluster-Based Client Selection for Dependent Multi-Task Federated Learning in Edge Computing
Department of Statistics, University of Oxford（牛津大学统计系）；Guangdong Prov./Zhuhai Key Lab of IRADS, Beijing Normal-Hong Kong Baptist University（广东/珠海IRADS重点实验室，北京师范大学-香港 Baptist大学）；Institute of Precision Medicine, The First Affiliated Hospital, Sun Yat-Sen University（精准医学研究院，中山大学第一附属医院）
Baidu Inc.（百度公司）；Auburn University（奥本大学）；University of Oregon（俄勒冈大学）
In recent years, data and computing resources are typically distributed in the devices of end users, various regions or organizations. Because of laws or regulations, the distributed data and computing resources cannot be directly shared among different regions or organizations for machine learning tasks. Federated learning emerges as an efficient approach to exploit distributed data and computing resources, so as to collaboratively train machine learning models, while obeying the laws and regulations and ensuring data security and data privacy. In this paper, we provide a comprehensive survey…
Pervasive computing promotes the installation of connected devices in our living spaces in order to provide services. Two major developments have gained significant momentum recently: an advanced use of edge resources and the integration of machine learning techniques for engineering applications. This evolution raises major challenges, in particular related to the appropriate distribution of computing elements along an edge-to-cloud continuum. About this, Federated Learning has been recently proposed for distributed model training in the edge. The principle of this approach is to aggregate mo…
GQFedWAvg: Optimization-Based Quantized Federated Learning in General Edge Computing Systems
School of EIEE, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；IoT Thrust, the HKUST (GZ)（香港科技大学（广州）物联网枢纽）；Department of ECE, the HKUST（香港科技大学电子与计算机工程学系）
With the fast development of quantum computing and deep learning, quantum neural networks have attracted great attention recently. By leveraging the power of quantum computing, deep neural networks can potentially overcome computational power limitations in classic machine learning. However, when multiple quantum machines wish to train a global model using the local data on each machine, it may be very difficult to copy the data into one machine and train the model. Therefore, a collaborative quantum neural network framework is necessary. In this article, we borrow the core idea of federated l…
Mobile Edge Computing (MEC) has been a promising paradigm for communicating and edge processing of data on the move. We aim to employ Federated Learning (FL) and prominent features of blockchain into MEC architecture such as connected autonomous vehicles to enable complete decentralization, immutability, and rewarding mechanisms simultaneously. FL is advantageous for mobile devices with constrained connectivity since it requires model updates to be delivered to a central point instead of substantial amounts of data communication. For instance, FL in autonomous, connected vehicles can increase …
