The challenge of communication-efficient distributed optimization has attracted attention in recent years. In this paper, a communication efficient algorithm, called ordering-based alternating direction method of multipliers (OADMM) is devised in a general fully decentralized network setting where a worker can only exchange messages with neighbors. Compared to the classical ADMM, a key feature of OADMM is that transmissions are ordered among workers at each iteration such that a worker with the most informative data broadcasts its local variable to neighbors first, and neighbors who have not transmitted yet can update their local variables based on that received transmission. In OADMM, we prohibit workers from transmitting if their current local variables are not sufficiently different from their previously transmitted value. A variant of OADMM, called SOADMM, is proposed where transmissions are ordered but transmissions are never stopped for each node at each iteration. Numerical results demonstrate that given a targeted accuracy, OADMM can significantly reduce the number of communications compared to existing algorithms including ADMM. We also show numerically that SOADMM can accelerate convergence, resulting in communication savings compared to the classical ADMM.
ITP Lab, Department of EEE（ITP实验室，电子与电机工程系）；Imperial College London（帝国理工学院）
Federated learning has shown its advances over the last few years but is facing many challenges, such as how algorithms save communication resources, how they reduce computational costs, and whether they converge. To address these issues, this paper proposes exact and inexact ADMM-based federated learning. They are not only communication-efficient but also converge linearly under very mild conditions, such as convexity-free and irrelevance to data distributions. Moreover, the inexact version has low computational complexity, thereby alleviating the computational burdens significantly.
Singapore University of Technology and Design（新加坡科技设计大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sichuan University（四川大学）；The Hong Kong Polytechnic University（香港理工大学）
As a novel distributed learning paradigm, federated learning (FL) faces serious challenges in dealing with massive clients with heterogeneous data distribution and computation and communication resources. Various client-variance-reduction schemes and client sampling strategies have been respectively introduced to improve the robustness of FL. Among others, primal-dual algorithms such as the alternating direction of method multipliers (ADMM) have been found being resilient to data distribution and outperform most of the primal-only FL algorithms. However, the reason behind remains a mystery sti…
On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；La Trobe Business School, La Trobe University（拉筹伯大学拉筹伯商学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）；Hong Kong University of Science and Technology（香港科技大学）
Argonne National Laboratory（阿贡国家实验室）；Mathematics and Computer Science Division（数学与计算机科学部）；Lemont, IL 60439（美国伊利诺伊州莱蒙特市（60439））
Differential privacy (DP) techniques can be applied to the federated learning model to protect data privacy against inference attacks to communication among the learning agents. The DP techniques, however, hinder achieving a greater learning performance while ensuring strong data privacy. In this paper we develop a DP inexact alternating direction method of multipliers algorithm that solves a sequence of subproblems with the objective perturbation by random noises generated from a Laplace distribution. We show that our algorithm provides $\barε$-DP for every iteration, where $\barε$ is a priva…
Differential privacy (DP) techniques can be applied to the federated learning model to statistically guarantee data privacy against inference attacks to communication among the learning agents. While ensuring strong data privacy, however, the DP techniques hinder achieving a greater learning performance. In this paper we develop a DP inexact alternating direction method of multipliers algorithm with multiple local updates for federated learning, where a sequence of convex subproblems is solved with the objective perturbation by random noises generated from a Laplace distribution. We show that …
University of Science and Technology of China（中国科学技术大学）；Nanjing University of Science and Technology（南京理工大学）
Federated learning (FL) supports training models on geographically distributed devices. However, traditional FL systems adopt a centralized synchronous strategy, putting high communication pressure and model generalization challenge. Existing optimizations on FL either fail to speedup training on heterogeneous devices or suffer from poor communication efficiency. In this paper, we propose HADFL, a framework that supports decentralized asynchronous training on heterogeneous devices. The devices train model locally with heterogeneity-aware local steps using local data. In each aggregation cycle,…
By leveraging deep learning based technologies, the data-driven based approaches have reached great success with the rapid increase of data generated of Industrial Indernet of Things(IIot). However, security and privacy concerns are obstacles for data providers in many sensitive data-driven industrial scenarios, such as healthcare and auto-driving. Many Federated Learning(FL) approaches have been proposed with DNNs for IIoT applications, these works still suffer from low usability of data due to data incompleteness, low quality, insufficient quantity, sensitivity, etc. Therefore, we propose a …
Royal Military College of Canada（加拿大皇家军事学院）；Universidad Nacional Autónoma de México（墨西哥国立自治大学）
Federated Learning (FL) is a machine learning technique that addresses the privacy challenges in terms of access rights of local datasets by enabling the training of a model across nodes holding their data samples locally. To achieve decentralized federated learning, blockchain-based FL was proposed as a distributed FL architecture. In decentralized FL, the chief is eliminated from the learning process as workers collaborate between each other to train the global model. Decentralized FL applications need to account for the additional delay incurred by blockchain-based FL deployments. Particula…
Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experimen…
Inner Mongolia University of Technology（内蒙古工业大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；New Jersey Institute of Technology（新泽西理工学院）
Advanced researches on connected vehicles have recently targeted to the integration of vehicle-to-everything (V2X) networks with Machine Learning (ML) tools and distributed decision making. Federated learning (FL) is emerging as a new paradigm to train machine learning (ML) models in distributed systems, including vehicles in V2X networks. Rather than sharing and uploading the training data to the server, the updating of model parameters (e.g., neural networks' weights and biases) is applied by large populations of interconnected vehicles, acting as local learners. Despite these benefits, the …
Decentralized Federated Learning: A Survey on Security and Privacy
University of Windsor（温莎大学）；University of New Brunswick（新不伦瑞克大学）；University of Western Ontario（西安大略大学）；Hong Kong University of Science and Technology（香港科技大学）；WeBank（微众银行）
TrustChain: A Blockchain Framework for Auditing and Verifying Aggregators in Decentralized Federated Learning
Department of Electrical and Computer Engineering, University of Windsor（威省大学电气与计算机工程系）；Faculty of Computer Science, University of New Brunswick（新不伦瑞克大学计算机科学学院）
Federated learning opens a number of research opportunities due to its high communication efficiency in distributed training problems within a star network. In this paper, we focus on improving the communication efficiency for fully decentralized federated learning over a graph, where the algorithm performs local updates for several iterations and then enables communications among the nodes. In such a way, the communication rounds of exchanging the common interest of parameters can be saved significantly without loss of optimality of the solutions. Multiple numerical simulations based on large…
Federated Bayesian learning offers a principled framework for the definition of collaborative training algorithms that are able to quantify epistemic uncertainty and to produce trustworthy decisions. Upon the completion of collaborative training, an agent may decide to exercise her legal "right to be forgotten", which calls for her contribution to the jointly trained model to be deleted and discarded. This paper studies federated learning and unlearning in a decentralized network within a Bayesian framework. It specifically develops federated variational inference (VI) solutions based on the d…
Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration
College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）
Polytechnic Institute, Zhejiang University（浙江大学理工学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Public Affairs, Zhejiang University（浙江大学公共管理学院）
Federated learning(FL) is a rapidly growing field and many centralized and decentralized FL frameworks have been proposed. However, it is of great challenge for current FL frameworks to improve communication performance and maintain the security and robustness under malicious node attacks. In this paper, we propose Galaxy Federated Learning Framework(GFL), a decentralized FL framework based on blockchain. GFL introduces the consistent hashing algorithm to improve communication performance and proposes a novel ring decentralized FL algorithm(RDFL) to improve decentralized FL performance and ban…
Decentralized Learning (DL) is a peer--to--peer learning approach that allows a group of users to jointly train a machine learning model. To ensure correctness, DL should be robust, i.e., Byzantine users must not be able to tamper with the result of the collaboration. In this paper, we introduce two \textit{new} attacks against DL where a Byzantine user can: make the network converge to an arbitrary model of their choice, and exclude an arbitrary user from the learning process. We demonstrate our attacks' efficiency against Self--Centered Clipping, the state--of--the--art robust DL protocol. F…
Decentralized stochastic gradient descent (SGD) is a driving engine for decentralized federated learning (DFL). The performance of decentralized SGD is jointly influenced by inter-node communications and local updates. In this paper, we propose a general DFL framework, which implements both multiple local updates and multiple inter-node communications periodically, to strike a balance between communication efficiency and model consensus. It can provide a general decentralized SGD analytical framework. We establish strong convergence guarantees for the proposed DFL algorithm without the assumpt…
RISE Research Institutes of Sweden（瑞典RISE研究院）；Chalmers University of Technology（查尔姆斯理工大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
We tackle the non-convex problem of learning a personalized deep learning model in a decentralized setting. More specifically, we study decentralized federated learning, a peer-to-peer setting where data is distributed among many clients and where there is no central server to orchestrate the training. In real world scenarios, the data distributions are often heterogeneous between clients. Therefore, in this work we study the problem of how to efficiently learn a model in a peer-to-peer system with non-iid client data. We propose a method named Performance-Based Neighbor Selection (PENS) where…
University of Murcia（穆尔西亚大学）；University of Zurich（苏黎世大学）；armasuisse Science & Technology（瑞士联邦国防采购局科技部门）
Federated learning (FL) allows participants to collaboratively train machine and deep learning models while protecting data privacy. However, the FL paradigm still presents drawbacks affecting its trustworthiness since malicious participants could launch adversarial attacks against the training process. Related work has studied the robustness of horizontal FL scenarios under different attacks. However, there is a lack of work evaluating the robustness of decentralized vertical FL and comparing it with horizontal FL architectures affected by adversarial attacks. Thus, this work proposes three d…
The rapid expansion of data worldwide invites the need for more distributed solutions in order to apply machine learning on a much wider scale. The resultant distributed learning systems can have various degrees of centralization. In this work, we demonstrate our solution FLoBC for building a generic decentralized federated learning system using blockchain technology, accommodating any machine learning model that is compatible with gradient descent optimization. We present our system design comprising the two decentralized actors: trainer and validator, alongside our methodology for ensuring r…
Sun Yat-sen University（中山大学）；WeBank Co. Ltd.（微众银行股份有限公司）
Federated learning has been widely studied and applied to various scenarios. In mobile computing scenarios, federated learning protects users from exposing their private data, while cooperatively training the global model for a variety of real-world applications. However, the security of federated learning is increasingly being questioned, due to the malicious clients or central servers' constant attack to the global model or user privacy data. To address these security issues, we proposed a decentralized federated learning framework based on blockchain, i.e., a Blockchain-based Federated Lear…
Noise-Robust and Resource-Efficient ADMM-based Federated Learning
Department of Electronic Systems, Norwegian University of Science and Technology（挪威科学技术大学电子系统系）；Department of Information and Communications Engineering, Aalto University（阿alto大学信息与通信工程系）；Commonwealth Scientific and Industrial Research Organisation（共同科学与工业研究组织）；SDU Applied AI and Data Science, The Maersk Mc-Kinney Moller Institute, Faculty of Engineering, University of Southern Denmark（南部丹麦大学应用人工智能与数据科学，马士基麦金尼莫勒研究所，工程学院）
Iowa State University（爱荷华州立大学）；The Ohio State University（俄亥俄州立大学）
Federated learning (FL) has received a surge of interest in recent years thanks to its benefits in data privacy protection, efficient communication, and parallel data processing. Also, with appropriate algorithmic designs, one could achieve the desirable linear speedup for convergence effect in FL. However, most existing works on FL are limited to systems with i.i.d. data and centralized parameter servers and results on decentralized FL with heterogeneous datasets remains limited. Moreover, whether or not the linear speedup for convergence is achievable under fully decentralized FL with data h…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
Establishing how a set of learners can provide privacy-preserving federated learning in a fully decentralized (peer-to-peer, no coordinator) manner is an open problem. We propose the first privacy-preserving consensus-based algorithm for the distributed learners to achieve decentralized global model aggregation in an environment of high mobility, where the communication graph between the learners may vary between successive rounds of model aggregation. In particular, in each round of global model aggregation, the Metropolis-Hastings method is applied to update the weighted adjacency matrix bas…
University of Science and Technology of China（中国科学技术大学）；JD Explore Academy（京东探索研究院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Institute of Artificial Intelligence（人工智能研究院）
Personalized federated learning is proposed to handle the data heterogeneity problem amongst clients by learning dedicated tailored local models for each user. However, existing works are often built in a centralized way, leading to high communication pressure and high vulnerability when a failure or an attack on the central server occurs. In this work, we propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL, which employs personalized sparse masks to customize sparse local models on the edge. To further save the commu…
