Federated Learning Beyond the Star: Local D2D Model Consensus with Global Cluster Sampling
Purdue University（普渡大学）；Arizona State University（亚利桑那州立大学）
Frank Po-Chen Lin, Seyyedali Hosseinalipour, Sheikh Shams Azam, Christopher G. Brinton, Nicolò Michelusi
备注：
This paper has been published in IEEE Global Communications Conference 2021 (Globecom). arXiv admin note: substantial text overlap with arXiv:2103.10481
Federated learning has emerged as a popular technique for distributing model training across the network edge. Its learning architecture is conventionally a star topology between the devices and a central server. In this paper, we propose two timescale hybrid federated learning (TT-HF), which migrates to a more distributed topology via device-to-device (D2D) communications. In TT-HF, local model training occurs at devices via successive gradient iterations, and the synchronization process occurs at two timescales: (i) macro-scale, where global aggregations are carried out via device-server interactions, and (ii) micro-scale, where local aggregations are carried out via D2D cooperative consensus formation in different device clusters. Our theoretical analysis reveals how device, cluster, and network-level parameters affect the convergence of TT-HF, and leads to a set of conditions under which a convergence rate of O(1/t) is guaranteed. Experimental results demonstrate the improvements in convergence and utilization that can be obtained by TT-HF over state-of-the-art federated learning baselines.
Sun Yat-sen University（中山大学）；WeBank Co. Ltd.（微众银行股份有限公司）
Federated learning has been widely studied and applied to various scenarios. In mobile computing scenarios, federated learning protects users from exposing their private data, while cooperatively training the global model for a variety of real-world applications. However, the security of federated learning is increasingly being questioned, due to the malicious clients or central servers' constant attack to the global model or user privacy data. To address these security issues, we proposed a decentralized federated learning framework based on blockchain, i.e., a Blockchain-based Federated Lear…
University of Maryland, College Park（马里兰大学帕克分校）；University of British Columbia（不列颠哥伦比亚大学）；JP Morgan Chase AI Research（摩根大通人工智能研究院）；Amazon（亚马逊）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）
In this work, we quantitatively calibrate the performance of global and local models in federated learning through a multi-criterion optimization-based framework, which we cast as a constrained program. The objective of a device is its local objective, which it seeks to minimize while satisfying nonlinear constraints that quantify the proximity between the local and the global model. By considering the Lagrangian relaxation of this problem, we develop a novel primal-dual method called Federated Learning Beyond Consensus (\texttt{FedBC}). Theoretically, we establish that \texttt{FedBC} converge…
Inner Mongolia University of Technology（内蒙古工业大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；New Jersey Institute of Technology（新泽西理工学院）
Advanced researches on connected vehicles have recently targeted to the integration of vehicle-to-everything (V2X) networks with Machine Learning (ML) tools and distributed decision making. Federated learning (FL) is emerging as a new paradigm to train machine learning (ML) models in distributed systems, including vehicles in V2X networks. Rather than sharing and uploading the training data to the server, the updating of model parameters (e.g., neural networks' weights and biases) is applied by large populations of interconnected vehicles, acting as local learners. Despite these benefits, the …
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Federated learning involves a mixture of centralized and decentralized processing tasks, where a server regularly selects a sample of the agents and these in turn sample their local data to compute stochastic gradients for their learning updates. This process runs continually. The sampling of both agents and data is generally uniform; however, in this work we consider non-uniform sampling. We derive optimal importance sampling strategies for both agent and data selection and show that non-uniform sampling without replacement improves the performance of the original FedAvg algorithm. We run exp…
This paper proposes a fully decentralized federated learning (FL) scheme for Internet of Everything (IoE) devices that are connected via multi-hop networks. Because FL algorithms hardly converge the parameters of machine learning (ML) models, this paper focuses on the convergence of ML models in function spaces. Considering that the representative loss functions of ML tasks e.g, mean squared error (MSE) and Kullback-Leibler (KL) divergence, are convex functionals, algorithms that directly update functions in function spaces could converge to the optimal solution. The key concept of this paper …
Deakin University（迪肯大学）；Commonwealth Scientific and Industrial Research Organization（澳大利亚联邦科学与工业研究组织）；Xidian University（西安电子科技大学）；Qilu University of Technology（齐鲁工业大学）；Shandong Computer Science Center（山东省计算中心）
Since the traffic conditions change over time, machine learning models that predict traffic flows must be updated continuously and efficiently in smart public transportation. Federated learning (FL) is a distributed machine learning scheme that allows buses to receive model updates without waiting for model training on the cloud. However, FL is vulnerable to poisoning or DDoS attacks since buses travel in public. Some work introduces blockchain to improve reliability, but the additional latency from the consensus process reduces the efficiency of FL. Asynchronous Federated Learning (AFL) is a …
University of British Columbia（不列颠哥伦比亚大学）；Purdue University（普渡大学）；Tianjin University（天津大学）
Federated learning (FL) is an effective technique to directly involve edge devices in machine learning training while preserving client privacy. However, the substantial communication overhead of FL makes training challenging when edge devices have limited network bandwidth. Existing work to optimize FL bandwidth overlooks downstream transmission and does not account for FL client sampling. In this paper we propose GlueFL, a framework that incorporates new client sampling and model compression algorithms to mitigate low download bandwidths of FL clients. GlueFL prioritizes recently used client…
University of Delaware（特拉华大学）；Deakin University（迪肯大学）；University of Oulu（奥卢大学）
Federated learning (FL) is a promising distributed learning solution that only exchanges model parameters without revealing raw data. However, the centralized architecture of FL is vulnerable to the single point of failure. In addition, FL does not examine the legitimacy of local models, so even a small fraction of malicious devices can disrupt global training. To resolve these robustness issues of FL, in this paper, we propose a blockchain-based decentralized FL framework, termed VBFL, by exploiting two mechanisms in a blockchained architecture. First, we introduced a novel decentralized vali…
Consiglio Nazionale delle Ricerche（意大利国家研究委员会）；Insitute of Electronics, Computer and Telecommunication Engineering（电子、计算机与电信工程研究所）；Politecnico di Milano（米兰理工大学）
Federated learning (FL) is emerging as a new paradigm to train machine learning models in distributed systems. Rather than sharing, and disclosing, the training dataset with the server, the model parameters (e.g. neural networks weights and biases) are optimized collectively by large populations of interconnected devices, acting as local learners. FL can be applied to power-constrained IoT devices with slow and sporadic connections. In addition, it does not need data to be exported to third parties, preserving privacy. Despite these benefits, a main limit of existing approaches is the centrali…
Expediting In-Network Federated Learning by Voting-Based Consensus Model Compression
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Computing, Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院计算学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen University（深圳大学广东省人工智能与数字经济实验室（深圳））；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
While client sampling is a central operation of current state-of-the-art federated learning (FL) approaches, the impact of this procedure on the convergence and speed of FL remains under-investigated. In this work, we provide a general theoretical framework to quantify the impact of a client sampling scheme and of the clients heterogeneity on the federated optimization. First, we provide a unified theoretical ground for previously reported sampling schemes experimental results on the relationship between FL convergence and the variance of the aggregation weights. Second, we prove for the first…
Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Imperial College London（伦敦帝国理工学院）
Intermittent connectivity of clients to the parameter server (PS) is a major bottleneck in federated edge learning frameworks. The lack of constant connectivity induces a large generalization gap, especially when the local data distribution amongst clients exhibits heterogeneity. To overcome intermittent communication outages between clients and the central PS, we introduce the concept of collaborative relaying wherein the participating clients relay their neighbors' local updates to the PS in order to boost the participation of clients with poor connectivity to the PS. We propose a semi-decen…
ShanghaiTech University（上海科技大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we consider decentralized federated learning (FL) over wireless networks, where over-the-air computation (AirComp) is adopted to facilitate the local model consensus in a device-to-device (D2D) communication manner. However, the AirComp-based consensus phase brings the additive noise in each algorithm iterate and the consensus needs to be robust to wireless network topology changes, which introduce a coupled and novel challenge of establishing the convergence for wireless decentralized FL algorithm. To facilitate consensus phase, we propose an AirComp-based DSGD with gradient tr…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Federated learning encapsulates distributed learning strategies that are managed by a central unit. Since it relies on using a selected number of agents at each iteration, and since each agent, in turn, taps into its local data, it is only natural to study optimal sampling policies for selecting agents and their data in federated learning implementations. Usually, only uniform sampling schemes are used. However, in this work, we examine the effect of importance sampling and devise schemes for sampling agents and data non-uniformly guided by a performance measure. We find that in schemes involv…
Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT
School of Electronic and Optical Engineering, Nanjing University of Science and Technology（南京理工大学电子工程与光电技术学院）；School of Information Science and Engineering, Southeast University（东南大学信息科学与工程学院）；School of Computer Engineering, Jiangsu Second Normal University（江苏第二师范学院计算机工程学院）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Department of Electronics Engineering, Shanghai Jiao Tong University（上海交通大学电子工程系）；Digital Research Institute, ENN Group（新奥集团数字研究院）
Aalto University（阿尔托大学）；Columbia University（哥伦比亚大学）；Nokia Bell Labs（诺基亚贝尔实验室）；The University of Queensland（昆士兰大学）
Federated learning (FL) is a novel machine learning setting that enables on-device intelligence via decentralized training and federated optimization. Deep neural networks' rapid development facilitates the learning techniques for modeling complex problems and emerges into federated deep learning under the federated setting. However, the tremendous amount of model parameters burdens the communication network with a high load of transportation. This paper introduces two approaches for improving communication efficiency by dynamic sampling and top-$k$ selective masking. The former controls the f…
Quantum Cloud and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia（量子云与分布式系统实验室，计算机与信息系统学院，墨尔本大学，澳大利亚）；Department of Computer Science and Engineering, University of Visvesvaraya College of Engineering, Bangalore University, India（计算机科学与工程系，维萨瓦拉亚工程学院，班加罗尔大学，印度）
Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization
University of Wisconsin-Madison, Department of Statistics（威斯康星大学麦迪逊分校统计学系）；Google（谷歌）；University of Oxford, Mathematical Institute（牛津大学数学学院）；University of California, Los Angeles, Department of Statistics and Data Science（加州大学洛杉矶分校统计学与数据科学系）
Consiglio Nazionale delle Ricerche (CNR)（意大利国家研究委员会）；University of Oulu（奥卢大学）
Recent advances in distributed learning raise environmental concerns due to the large energy needed to train and move data to/from data centers. Novel paradigms, such as federated learning (FL), are suitable for decentralized model training across devices or silos that simultaneously act as both data producers and learners. Unlike centralized learning (CL) techniques, relying on big-data fusion and analytics located in energy hungry data centers, in FL scenarios devices collaboratively train their models without sharing their private data. This article breaks down and analyzes the main factors…
Establishing how a set of learners can provide privacy-preserving federated learning in a fully decentralized (peer-to-peer, no coordinator) manner is an open problem. We propose the first privacy-preserving consensus-based algorithm for the distributed learners to achieve decentralized global model aggregation in an environment of high mobility, where the communication graph between the learners may vary between successive rounds of model aggregation. In particular, in each round of global model aggregation, the Metropolis-Hastings method is applied to update the weighted adjacency matrix bas…
University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
It is well understood that client-master communication can be a primary bottleneck in Federated Learning. In this work, we address this issue with a novel client subsampling scheme, where we restrict the number of clients allowed to communicate their updates back to the master node. In each communication round, all participating clients compute their updates, but only the ones with "important" updates communicate back to the master. We show that importance can be measured using only the norm of the update and give a formula for optimal client participation. This formula minimizes the distance …
Federated learning (FL) is an emerging collaborative machine learning method to train models on distributed datasets with privacy concerns. To properly incentivize data owners to contribute their efforts, Shapley Value (SV) is often adopted to fairly assess their contribution. However, the calculation of SV is time-consuming and computationally costly. In this paper, we propose FedCoin, a blockchain-based peer-to-peer payment system for FL to enable a feasible SV based profit distribution. In FedCoin, blockchain consensus entities calculate SVs and a new block is created based on the proof of …
FedSampling: A Better Sampling Strategy for Federated Learning
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）；Sony AI（索尼人工智能）；Zhongguancun Laboratory（中关村实验室）；Institute for Precision Medicine of Tsinghua University（清华大学精准医学研究院）
A recent emphasis of distributed learning research has been on federated learning (FL), in which model training is conducted by the data-collecting devices. Existing research on FL has mostly focused on a star topology learning architecture with synchronized (time-triggered) model training rounds, where the local models of the devices are periodically aggregated by a centralized coordinating node. However, in many settings, such a coordinating node may not exist, motivating efforts to fully decentralize FL. In this work, we propose a novel methodology for distributed model aggregations via asy…
Towards Serverless Semi-Decentralized Federated Learning with Heterogeneous Optimizers
Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电子工程与计算机科学系）；Department of Electrical and Computer Engineering, Purdue University（帕克森大学电子工程与计算机科学系）
Federated learning uses a set of techniques to efficiently distribute the training of a machine learning algorithm across several devices, who own the training data. These techniques critically rely on reducing the communication cost -- the main bottleneck -- between the devices and a central server. Federated learning algorithms usually take an optimization approach: they are algorithms for minimizing the training loss subject to communication (and other) constraints. In this work, we instead take a Bayesian approach for the training task, and propose a communication-efficient variant of the …
Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedback
University of Chicago Booth School of Business（芝加哥大学布斯商学院）；New Jersey Institute of Technology（新泽西理工学院）；Ant Financial Group（蚂蚁集团）；College of Computer Science and Technology Zhejiang University（浙江大学计算机科学与技术学院）；University of Southern California Marshall School of Business（南加州大学马歇尔商学院）
Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
University of Electronic Science and Technology of China（电子科技大学）；Fudan University（复旦大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Meta AI；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
The Hong Kong University of Science and Technology（香港科技大学）
Federated learning (FL) is a promising paradigm to enable privacy-preserving deep learning from distributed data. Most previous works are based on federated average (FedAvg), which, however, faces several critical issues, including a high communication overhead and the difficulty in dealing with heterogeneous model architectures. Federated Distillation (FD) is a recently proposed alternative to enable communication-efficient and robust FL, which achieves orders of magnitude reduction of the communication overhead compared with FedAvg and is flexible to handle heterogeneous models at the client…
This work addresses the problem of optimizing communications between server and clients in federated learning (FL). Current sampling approaches in FL are either biased, or non optimal in terms of server-clients communications and training stability. To overcome this issue, we introduce \textit{clustered sampling} for clients selection. We prove that clustered sampling leads to better clients representatitivity and to reduced variance of the clients stochastic aggregation weights in FL. Compatibly with our theory, we provide two different clustering approaches enabling clients aggregation based…
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL …
DELTA: Diverse Client Sampling for Fasting Federated Learning
School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）；The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；School of Engineering, Westlake University（西湖大学工学院）；The Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来智能网络重点实验室）
Bayesian Network Structural Consensus via Greedy Min-Cut Analysis
Instituto de Investigación en Informática de Albacete (I3A), Universidad de Castilla-La Mancha, Albacete, Spain（阿尔巴切特信息研究所（I3A）、卡斯蒂利亚-拉曼查大学，阿尔巴切特，西班牙）；Departamento de Sistemas Informáticos, Universidad de Castilla-La Mancha, Albacete, Spain（信息系统系、卡斯蒂利亚-拉曼查大学，阿尔巴切特，西班牙）；Departamento de Matemáticas, Universidad de Castilla-La Mancha, Albacete, Spain（数学系、卡斯蒂利亚-拉曼查大学，阿尔巴切特，西班牙）
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Yale University（耶鲁大学）
Federated learning (FL) algorithms usually sample a fraction of clients in each round (partial participation) when the number of participants is large and the server's communication bandwidth is limited. Recent works on the convergence analysis of FL have focused on unbiased client sampling, e.g., sampling uniformly at random, which suffers from slow wall-clock time for convergence due to high degrees of system heterogeneity and statistical heterogeneity. This paper aims to design an adaptive client sampling algorithm that tackles both system and statistical heterogeneity to minimize the wall-…
SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；The Hong Kong Polytechnic University（香港理工大学）；Guangdong University of Technology（广东工业大学）；Southeast University（东南大学）
