Federated learning (FL) aims to train machine learning (ML) models collaboratively using decentralized data, bypassing the need for centralized data aggregation. Standard FL models often assume that all data come from the same unknown distribution. However, in practical situations, decentralized data frequently exhibit heterogeneity. We propose a novel FL model, Distributionally Robust Federated Learning (DRFL), that applies distributionally robust optimization to overcome the challenges posed by data heterogeneity and distributional ambiguity. We derive a tractable reformulation for DRFL and develop a novel solution method based on the alternating direction method of multipliers (ADMM) algorithm to solve this problem. Our experimental results demonstrate that DRFL outperforms standard FL models under data heterogeneity and ambiguity.
ITP Lab, Department of EEE（ITP实验室，电子与电机工程系）；Imperial College London（帝国理工学院）
Federated learning has shown its advances over the last few years but is facing many challenges, such as how algorithms save communication resources, how they reduce computational costs, and whether they converge. To address these issues, this paper proposes exact and inexact ADMM-based federated learning. They are not only communication-efficient but also converge linearly under very mild conditions, such as convexity-free and irrelevance to data distributions. Moreover, the inexact version has low computational complexity, thereby alleviating the computational burdens significantly.
Singapore University of Technology and Design（新加坡科技设计大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sichuan University（四川大学）；The Hong Kong Polytechnic University（香港理工大学）
As a novel distributed learning paradigm, federated learning (FL) faces serious challenges in dealing with massive clients with heterogeneous data distribution and computation and communication resources. Various client-variance-reduction schemes and client sampling strategies have been respectively introduced to improve the robustness of FL. Among others, primal-dual algorithms such as the alternating direction of method multipliers (ADMM) have been found being resilient to data distribution and outperform most of the primal-only FL algorithms. However, the reason behind remains a mystery sti…
Argonne National Laboratory（阿贡国家实验室）；Mathematics and Computer Science Division（数学与计算机科学部）；Lemont, IL 60439（美国伊利诺伊州莱蒙特市（60439））
Differential privacy (DP) techniques can be applied to the federated learning model to protect data privacy against inference attacks to communication among the learning agents. The DP techniques, however, hinder achieving a greater learning performance while ensuring strong data privacy. In this paper we develop a DP inexact alternating direction method of multipliers algorithm that solves a sequence of subproblems with the objective perturbation by random noises generated from a Laplace distribution. We show that our algorithm provides $\barε$-DP for every iteration, where $\barε$ is a priva…
Differential privacy (DP) techniques can be applied to the federated learning model to statistically guarantee data privacy against inference attacks to communication among the learning agents. While ensuring strong data privacy, however, the DP techniques hinder achieving a greater learning performance. In this paper we develop a DP inexact alternating direction method of multipliers algorithm with multiple local updates for federated learning, where a sequence of convex subproblems is solved with the objective perturbation by random noises generated from a Laplace distribution. We show that …
Distributionally Robust Federated Learning with Outlier Resilience
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（瑞典皇家理工学院电气工程与计算机科学系决策与控制系统 division）；Department of Control Science and Engineering, College of Electronics and Information Engineering, Tongji University（同济大学电子信息工程学院控制科学与工程系）；Mechanical Engineering and Material Science, Duke University（达特茅斯大学机械工程与材料科学系）
Tencent AI Lab（腾讯人工智能实验室）；Hong Kong University of Science and Technology（香港科技大学）
Recently, federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. Nevertheless, directly applying federated learning to real-world tasks faces two challenges: (1) heterogeneity in the data among different organizations; and (2) data noises inside individual organizations. In this paper, we propose a general framework to solve the above two challenges simultaneously. Specifically, we propose using distributionally robust optimization to mitigate the negative effects caused by data heterogeneity par…
Lehigh University（利哈伊大学）；Army Research Laboratory（陆军研究实验室）
The challenge of communication-efficient distributed optimization has attracted attention in recent years. In this paper, a communication efficient algorithm, called ordering-based alternating direction method of multipliers (OADMM) is devised in a general fully decentralized network setting where a worker can only exchange messages with neighbors. Compared to the classical ADMM, a key feature of OADMM is that transmissions are ordered among workers at each iteration such that a worker with the most informative data broadcasts its local variable to neighbors first, and neighbors who have not t…
FedAPM: Federated Learning via ADMM with Partial Model Personalization
School of Computer Science（计算机科学学院）；Wuhan University（武汉大学）；School of Management（管理学院）；Xi’an Jiaotong University（西安交通大学）；La Trobe Business School（拉筹伯商学院）；La Trobe University（拉筹伯大学）；Hong Kong University of Science and Technology（香港科技大学）；Oceanbase Ant group（Oceanbase蚂蚁集团）
On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；La Trobe Business School, La Trobe University（拉筹伯大学拉筹伯商学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）；Hong Kong University of Science and Technology（香港科技大学）
Noise-Robust and Resource-Efficient ADMM-based Federated Learning
Department of Electronic Systems, Norwegian University of Science and Technology（挪威科学技术大学电子系统系）；Department of Information and Communications Engineering, Aalto University（阿alto大学信息与通信工程系）；Commonwealth Scientific and Industrial Research Organisation（共同科学与工业研究组织）；SDU Applied AI and Data Science, The Maersk Mc-Kinney Moller Institute, Faculty of Engineering, University of Southern Denmark（南部丹麦大学应用人工智能与数据科学，马士基麦金尼莫勒研究所，工程学院）
Improving Privacy-Preserving Vertical Federated Learning by Efficient Communication with ADMM
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）；UC Berkeley（加州大学伯克利分校）；Amazon Web Services（亚马逊网络服务）；University of Chicago（芝加哥大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
针对纵向联邦学习中逐轮梯度通信成本高且消耗隐私预算的问题，提出多头 VIM 框架与 ADMM 方法，支持多次本地更新并提供用户级差分隐私，在四个数据集上显著提升性能与收敛速度。
Cooperative Intelligent Transport System (C-ITS) is a promising network to provide safety, efficiency, sustainability, and comfortable services for automated vehicles and road infrastructures by taking advantages from participants. However, the components of C-ITS usually generate large amounts of data, which makes it difficult to explore data science. Currently, federated learning has been proposed as an appealing approach to allow users to cooperatively reap the benefits from trained participants. Therefore, in this paper, we propose a novel Semi-asynchronous Hierarchical Federated Learning …
Wireless connectivity is instrumental in enabling scalable federated learning (FL), yet wireless channels bring challenges for model training, in which channel randomness perturbs each worker's model update while multiple workers' updates incur significant interference under limited bandwidth. To address these challenges, in this work we formulate a novel constrained optimization problem, and propose an FL framework harnessing wireless channel perturbations and interference for improving privacy, bandwidth-efficiency, and scalability. The resultant algorithm is coined analog federated ADMM (A-…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；University of Oulu（奥卢大学）；Purdue University（普渡大学）
Newton-type methods are popular in federated learning due to their fast convergence. Still, they suffer from two main issues, namely: low communication efficiency and low privacy due to the requirement of sending Hessian information from clients to parameter server (PS). In this work, we introduced a novel framework called FedNew in which there is no need to transmit Hessian information from clients to PS, hence resolving the bottleneck to improve communication efficiency. In addition, FedNew hides the gradient information and results in a privacy-preserving approach compared to the existing s…
TablePuppet: A Generic Framework for Relational Federated Learning
ETH Zürich（苏黎世联邦理工学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chicago（芝加哥大学）；Tsinghua University（清华大学）；Microsoft Research（微软研究院）
For distributed learning among collaborative users, this paper develops and analyzes a communication-efficient scheme for federated learning (FL) over the air, which incorporates 1-bit compressive sensing (CS) into analog aggregation transmissions. To facilitate design parameter optimization, we theoretically analyze the efficacy of the proposed scheme by deriving a closed-form expression for the expected convergence rate of the FL over the air. Our theoretical results reveal the tradeoff between convergence performance and communication efficiency as a result of the aggregation errors caused …
The University of Sydney（悉尼大学）；The Australian National University（澳大利亚国立大学）
In federated learning, participating clients typically possess non-i.i.d. data, posing a significant challenge to generalization to unseen distributions. To address this, we propose a Wasserstein distributionally robust optimization scheme called WAFL. Leveraging its duality, we frame WAFL as an empirical surrogate risk minimization problem, and solve it using a local SGD-based algorithm with convergence guarantees. We show that the robustness of WAFL is more general than related approaches, and the generalization bound is robust to all adversarial distributions inside the Wasserstein ball (am…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Michigan State University（密歇根州立大学）
Federated learning (FL) emerges as a popular distributed learning schema that learns a model from a set of participating users without sharing raw data. One major challenge of FL comes with heterogeneous users, who may have distributionally different (or non-iid) data and varying computation resources. As federated users would use the model for prediction, they often demand the trained model to be robust against malicious attackers at test time. Whereas adversarial training (AT) provides a sound solution for centralized learning, extending its usage for federated users has imposed significant …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Purdue University（普渡大学）；Microsoft（微软）；Nokia Bell Labs（诺基亚贝尔实验室）
Federated learning is a promising framework for learning over decentralized data spanning multiple regions. This approach avoids expensive central training data aggregation cost and can improve privacy because distributed sites do not have to reveal privacy-sensitive data. In this paper, we develop a privacy-preserving decentralized aggregation protocol for federated learning. We formulate the distributed aggregation protocol with the Alternating Direction Method of Multiplier (ADMM) and examine its privacy weakness. Unlike prior work that use Differential Privacy or homomorphic encryption for…
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Traditionally, Bayesian network structure learning is often carried out at a central site, in which all data is gathered. However, in practice, data may be distributed across different parties (e.g., companies, devices) who intend to collectively learn a Bayesian network, but are not willing to disclose information related to their data owing to privacy or security concerns. In this work, we present a federated learning approach to estimate the structure of Bayesian network from data that is horizontally partitioned across different parties. We develop a distributed structure learning method b…
In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can only be updated infrequently on the global stage. To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate…
Central South University（中南大学）；Arizona State University（亚利桑那州立大学）
In order to meet the requirements for performance, safety, and latency in many IoT applications, intelligent decisions must be made right here right now at the network edge. However, the constrained resources and limited local data amount pose significant challenges to the development of edge AI. To overcome these challenges, we explore continual edge learning capable of leveraging the knowledge transfer from previous tasks. Aiming to achieve fast and continual edge learning, we propose a platform-aided federated meta-learning architecture where edge nodes collaboratively learn a meta-model, a…
Centre for Wireless Communications (CWC), University of Oulu（奥卢大学无线通信中心（CWC））；Princeton University（普林斯顿大学）
In this article, we study the problem of robust reconfigurable intelligent surface (RIS)-aided downlink communication over heterogeneous RIS types in the supervised learning setting. By modeling downlink communication over heterogeneous RIS designs as different workers that learn how to optimize phase configurations in a distributed manner, we solve this distributed learning problem using a distributionally robust formulation in a communication-efficient manner, while establishing its rate of convergence. By doing so, we ensure that the global model performance of the worst-case worker is clos…
FDR-SVM: A Federated Distributionally Robust Support Vector Machine via a Mixture of Wasserstein Balls Ambiguity Set
H. Milton Stewart School of Industrial and Systems Engineering（H. Milton Stewart工业与系统工程学院）；Georgia Institute of Technology（佐治亚理工学院）；Department of Electrical Engineering（电气工程系）；University of Chile（智利大学）
This article proposes a communication-efficient decentralized deep learning algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an empirical risk, every worker in L-FGADMM periodically communicates with two neighbors, in which the periods are separately adjusted for different layers of its deep neural network. A constrained optimization problem for this setting is formulated and solved using the stochastic version of GADMM proposed in our prior work. Numerical evaluations show that by less frequently exchanging the largest layer, L-FGADMM can significantly reduce the comm…
University of Electronic Science and Technology of China（电子科技大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Sun Yat-Sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）
Federated learning (FL) is an emerging machine learning paradigm that allows to accomplish model training without aggregating data at a central server. Most studies on FL consider a centralized framework, in which a single server is endowed with a central authority to coordinate a number of devices to perform model training in an iterative manner. Due to stringent communication and bandwidth constraints, such a centralized framework has limited scalability as the number of devices grows. To address this issue, in this paper, we propose a ConFederated Learning (CFL) framework. The proposed CFL …
AugFL: Augmenting Federated Learning with Pretrained Models
School of Cyber Science and Technology, Sun Yat-sen University（中山大学网络空间安全学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Zhongguancun Laboratory（中关村实验室）；Department of Electrical and Computer Engineering, University of California, Davis（加州大学戴维斯分校电气与计算机工程系）
Federated learning, where algorithms are trained across multiple decentralized devices without sharing local data, is increasingly popular in distributed machine learning practice. Typically, a graph structure $G$ exists behind local devices for communication. In this work, we consider parameter estimation in federated learning with data distribution and communication heterogeneity, as well as limited computational capacity of local devices. We encode the distribution heterogeneity by parametrizing distributions on local devices with a set of distinct $p$-dimensional vectors. We then propose t…
