Federated Learning(FL) is a privacy-preserving machine learning paradigm where a global model is trained in-situ across a large number of distributed edge devices. These systems are often comprised of millions of user devices and only a subset of available devices can be used for training in each epoch. Designing a device selection strategy is challenging, given that devices are highly heterogeneous in both their system resources and training data. This heterogeneity makes device selection very crucial for timely model convergence and sufficient model accuracy. To tackle the FL client heterogeneity problem, various client selection algorithms have been developed, showing promising performance improvement in terms of model coverage and accuracy. In this work, we study the overhead of client selection algorithms in a large scale FL environment. Then we propose an efficient data distribution summary calculation algorithm to reduce the overhead in a real-world large scale FL environment. The evaluation shows that our proposed solution could achieve up to 30x reduction in data summary time, and up to 360x reduction in clustering time.
Koç University（科奇大学）；Imperial College London（伦敦帝国理工学院）
Federated learning (FL) has become de facto framework for collaborative learning among edge devices with privacy concern. The core of the FL strategy is the use of stochastic gradient descent (SGD) in a distributed manner. Large scale implementation of FL brings new challenges, such as the incorporation of acceleration techniques designed for SGD into the distributed setting, and mitigation of the drift problem due to non-homogeneous distribution of local datasets. These two problems have been separately studied in the literature; whereas, in this paper, we show that it is possible to address …
Federated learning (FL) is a fast-developing technique that allows multiple workers to train a global model based on a distributed dataset. Conventional FL (FedAvg) employs gradient descent algorithm, which may not be efficient enough. Momentum is able to improve the situation by adding an additional momentum step to accelerate the convergence and has demonstrated its benefits in both centralized and FL environments. It is well-known that Nesterov Accelerated Gradient (NAG) is a more advantageous form of momentum, but it is not clear how to quantify the benefits of NAG in FL so far. This motiv…
Auburn University（奥本大学）；Sony AI（索尼人工智能）；Baidu Research（百度研究院）；University of Oregon（俄勒冈大学）
The federated learning (FL) framework enables edge clients to collaboratively learn a shared inference model while keeping privacy of training data on clients. Recently, many heuristics efforts have been made to generalize centralized adaptive optimization methods, such as SGDM, Adam, AdaGrad, etc., to federated settings for improving convergence and accuracy. However, there is still a paucity of theoretical principles on where to and how to design and utilize adaptive optimization methods in federated settings. This work aims to develop novel adaptive optimization methods for FL from the pers…
Accelerated Training of Federated Learning via Second-Order Methods
Department of Artificial Intelligence, Indian Institute of Technology Hyderabad, India（人工智能系，印度理工学院海得拉尔分校）；Department of Computing Technologies, Swinburne University of Technology, Hawthorn, australia（计算技术系，斯威本科技大学 Hawthorn 分校）；Department of Mechanical Engineering , Indian Institute of Technology Delhi, India（机械工程系，印度理工学院德里分校）；Department of Computer Science and Engineering, Indian Institute of Technology Hyderabad, India（计算机科学与工程系，印度理工学院海得拉尔分校）
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
In this work, we consider the optimization formulation of personalized federated learning recently introduced by Hanzely and Richtárik (2020) which was shown to give an alternative explanation to the workings of local {\tt SGD} methods. Our first contribution is establishing the first lower bounds for this formulation, for both the communication complexity and the local oracle complexity. Our second contribution is the design of several optimal methods matching these lower bounds in almost all regimes. These are the first provably optimal methods for personalized federated learning. Our optima…
Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；Imaging Genetics Center, Stevens Neuroimaging and Informatics Institute, University of Southern California（南加州大学史蒂文斯神经影像与信息学研究所影像遗传学中心）
There are situations where data relevant to machine learning problems are distributed across multiple locations that cannot share the data due to regulatory, competitiveness, or privacy reasons. Machine learning approaches that require data to be copied to a single location are hampered by the challenges of data sharing. Federated Learning (FL) is a promising approach to learn a joint model over all the available data across silos. In many cases, the sites participating in a federation have different data distributions and computational capabilities. In these heterogeneous environments, existi…
Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI)（沙特数据与人工智能局-阿卜杜拉国王科技大学数据科学与人工智能卓越中心）
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Intel Labs（英特尔研究院）；University of Central Florida（中佛罗里达大学）
Federated learning (FL) is a distributed machine learning technology for next-generation AI systems that allows a number of workers, i.e., edge devices, collaboratively learn a shared global model while keeping their data locally to prevent privacy leakage. Enabling FL over wireless multi-hop networks can democratize AI and make it accessible in a cost-effective manner. However, the noisy bandwidth-limited multi-hop wireless connections can lead to delayed and nomadic model updates, which significantly slows down the FL convergence speed. To address such challenges, this paper aims to accelera…
Federated Learning (FL) has emerged as a de facto machine learning area and received rapid increasing research interests from the community. However, catastrophic forgetting caused by data heterogeneity and partial participation poses distinctive challenges for FL, which are detrimental to the performance. To tackle the problems, we propose a new FL approach (namely GradMA), which takes inspiration from continual learning to simultaneously correct the server-side and worker-side update directions as well as take full advantage of server's rich computing and memory resources. Furthermore, we el…
Update Estimation and Scheduling for Over-the-Air Federated Learning with Energy Harvesting Devices
Dept. of Electrical and Electronics Engineering, Bilkent University（电子与电子工程系，比尔肯特大学）；Dept. of Electrical and Electronic Engineering, Imperial College London（电子与电子工程系，帝国理工学院伦敦分校）
MammoFL: Mammographic Breast Density Estimation using Federated Learning
University of Pennsylvania（宾夕法尼亚大学）；Massachusetts Institute of Technology（麻省理工学院）；GSK（葛兰素史克）；Indiana University School of Medicine（印第安纳大学医学院）；Mayo Clinic（梅奥诊所）
Petabytes of data are generated each day by emerging Internet of Things (IoT), but only few of them can be finally collected and used for Machine Learning (ML) purposes due to the apprehension of data & privacy leakage, which seriously retarding ML's growth. To alleviate this problem, Federated learning is proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning introduces massive communication overhead as the synchronized data in each epoch is of the same size as the model, and thereby leading to a l…
School of Mathematical Sciences, Peking University（北京大学数学科学学院）
Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decentralized manner with high heterogeneity. This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency. We first establish a {\it functional central limit theorem} that shows the averaged iterates of Local SGD weakly converge to a rescaled B…
Federated learning (FL) is typically performed in a synchronous parallel manner, where the involvement of a slow client delays a training iteration. Current FL systems employ a participant selection strategy to select fast clients with quality data in each iteration. However, this is not always possible in practice, and the selection strategy often has to navigate an unpleasant trade-off between the speed and the data quality of clients. In this paper, we present Pisces, an asynchronous FL system with intelligent participant selection and model aggregation for accelerated training. To avoid in…
Federated learning (FL) is an emerging technique used to train a machine-learning model collaboratively using the data and computation resource of the mobile devices without exposing privacy-sensitive user data. Appropriate incentive mechanisms that motivate the data and mobile-device owner to participate in FL is key to building a sustainable platform for FL. However, it is difficult to evaluate the contribution level of the devices/owners to determine appropriate rewards without large computation and communication overhead. This paper proposes a computation-and communication-efficient method…
Quick global aggregation of effective distributed parameters is crucial to federated learning (FL), which requires adequate bandwidth for parameters communication and sufficient user data for local training. Otherwise, FL may cost excessive training time for convergence and produce inaccurate models. In this paper, we propose a brand-new FL framework, PromptFL, that replaces the federated model training with the federated prompt training, i.e., let federated participants train prompts instead of a shared model, to simultaneously achieve the efficient global aggregation and local training on in…
Robust Federated Personalised Mean Estimation for the Gaussian Mixture Model
School of Technology \& Comp. Sc. Tata Institute of Fundamental Research Mumbai, India；Department of Electrical Engineering University of California Los Angeles, USA
Northwestern Polytechnical University（西北工业大学）；Xidian University（西安电子科技大学）；University of Oslo（奥斯陆大学）；Simula Metropolitan Center for Digital Engineering（西穆拉都会数字工程中心）
Industrial Internet of Things (IoT) enables distributed intelligent services varying with the dynamic and realtime industrial devices to achieve Industry 4.0 benefits. In this paper, we consider a new architecture of digital twin empowered Industrial IoT where digital twins capture the characteristics of industrial devices to assist federated learning. Noticing that digital twins may bring estimation deviations from the actual value of device state, a trusted based aggregation is proposed in federated learning to alleviate the effects of such deviation. We adaptively adjust the aggregation fre…
VMware Research（威睿研究院）；University College London（伦敦大学学院）；Ben-Gurion University（本-古里安大学）；Stanford University（斯坦福大学）；Harvard University（哈佛大学）
Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this w…
Zhejiang University（浙江大学）；Sun Yat-sen University（中山大学）；Singapore University of Technology and Design（新加坡科技设计大学）
This paper proposes a client selection (CS) method to tackle the communication bottleneck of federated learning (FL) while concurrently coping with FL's data heterogeneity issue. Specifically, we first analyze the effect of CS in FL and show that FL training can be accelerated by adequately choosing participants to diversify the training dataset in each round of training. Based on this, we leverage data profiling and determinantal point process (DPP) sampling techniques to develop an algorithm termed Federated Learning with DPP-based Participant Selection (FL-DP$^3$S). This algorithm effective…
Southern University of Science and Technology（南方科技大学）；The University of Tokyo（东京大学）
Estimating the travel time of a path is an essential topic for intelligent transportation systems. It serves as the foundation for real-world applications, such as traffic monitoring, route planning, and taxi dispatching. However, building a model for such a data-driven task requires a large amount of users' travel information, which directly relates to their privacy and thus is less likely to be shared. The non-Independent and Identically Distributed (non-IID) trajectory data across data owners also make a predictive model extremely challenging to be personalized if we directly apply federate…
The Chinese University of Hong Kong（香港中文大学）；NVIDIA（英伟达）
How to ensure fairness is an important topic in federated learning (FL). Recent studies have investigated how to reward clients based on their contribution (collaboration fairness), and how to achieve uniformity of performance across clients (performance fairness). Despite achieving progress on either one, we argue that it is critical to consider them together, in order to engage and motivate more diverse clients joining FL to derive a high-quality global model. In this work, we propose a novel method to optimize both types of fairness simultaneously. Specifically, we propose to estimate clien…
As a distributed learning approach, federated learning trains a shared learning model over distributed datasets while preserving the training data privacy. We extend the application of federated learning to parking management and introduce FedParking in which Parking Lot Operators (PLOs) collaborate to train a long short-term memory model for parking space estimation without exchanging the raw data. Furthermore, we investigate the management of Parked Vehicle assisted Edge Computing (PVEC) by FedParking. In PVEC, different PLOs recruit PVs as edge computing nodes for offloading services throug…
Duzce University（杜兹杰大学）；University of Hertfordshire（赫特福德大学）；Koc University（科奇大学）
Machine learning (ML) has attracted a great research interest for physical layer design problems, such as channel estimation, thanks to its low complexity and robustness. Channel estimation via ML requires model training on a dataset, which usually includes the received pilot signals as input and channel data as output. In previous works, model training is mostly done via centralized learning (CL), where the whole training dataset is collected from the users at the base station (BS). This approach introduces huge communication overhead for data collection. In this paper, to address this challe…
Owen Sampling Accelerates Contribution Estimation in Federated Learning
Department of Electrical and Software Engineering, University of Calgary, Calgary, Alberta, Canada（电气与软件工程系，卡里尔大学）；Department of Electrical Engineering and Computer Science, York University, Toronto, Ontario, Canada（电气工程与计算机科学系，约克大学）；School of Information, University of Michigan, Ann Arbor, Michigan, USA（信息学院，密歇根大学）
Institute of Communication Acoustics, Ruhr-Universit\"at Bochum 44780 Bochum, Germany
In this paper we introduce a realistic and challenging, multi-source and multi-room acoustic environment and an improved algorithm for the estimation of source-dominated microphone clusters in acoustic sensor networks. Our proposed clustering method is based on a single microphone per node and on unsupervised clustered federated learning which employs a light-weight autoencoder model. We present an improved clustering control strategy that takes into account the variability of the acoustic scene and allows the estimation of a dynamic range of clusters using reduced amounts of training data. Th…
Massachusetts Institute of Technology（麻省理工学院）；Imperial College London（帝国理工学院）
Standard machine learning approaches require centralizing the users' data in one computer or a shared database, which raises data privacy and confidentiality concerns. Therefore, limiting central access is important, especially in healthcare settings, where data regulations are strict. A potential approach to tackling this is Federated Learning (FL), which enables multiple parties to collaboratively learn a shared prediction model by using parameters of locally trained models while keeping raw training data locally. In the context of AI-assisted pain-monitoring, we wish to enable confidentiali…
FedAds: A Benchmark for Privacy-Preserving CVR Estimation with Vertical Federated Learning
Alibaba Group（阿里巴巴集团）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
University of Bristol（布里斯托大学）；Toshiba Europe Ltd.（东芝欧洲有限公司）
Data-enabled cities are recently accelerated and enhanced with automated learning for improved Smart Cities applications. In the context of an Internet of Things (IoT) ecosystem, the data communication is frequently costly, inefficient, not scalable and lacks security. Federated Learning (FL) plays a pivotal role in providing privacy-preserving and communication efficient Machine Learning (ML) frameworks. In this paper we evaluate the feasibility of FL in the context of a Smart Cities Street Light Monitoring application. FL is evaluated against benchmarks of centralised and (fully) personalise…
