Initializing with pre-trained models when learning on downstream tasks is becoming standard practice in machine learning. Several recent works explore the benefits of pre-trained initialization in a federated learning (FL) setting, where the downstream training is performed at the edge clients with heterogeneous data distribution. These works show that starting from a pre-trained model can substantially reduce the adverse impact of data heterogeneity on the test performance of a model trained in a federated setting, with no changes to the standard FedAvg training algorithm. In this work, we provide a deeper theoretical understanding of this phenomenon. To do so, we study the class of two-layer convolutional neural networks (CNNs) and provide bounds on the training error convergence and test error of such a network trained with FedAvg. We introduce the notion of aligned and misaligned filters at initialization and show that the data heterogeneity only affects learning on misaligned filters. Starting with a pre-trained model typically results in fewer misaligned filters at initialization, thus producing a lower test error even when the model is trained in a federated setting with data heterogeneity. Experiments in synthetic settings and practical FL training on CNNs verify our theoretical findings.
An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to the fact that client devices have different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training t…
An oft-cited challenge of federated learning is the presence of heterogeneity. \emph{Data heterogeneity} refers to the fact that data from different clients may follow very different distributions. \emph{System heterogeneity} refers to client devices having different system capabilities. A considerable number of federated optimization methods address this challenge. In the literature, empirical evaluations usually start federated training from random initialization. However, in many practical applications of federated learning, the server has access to proxy data for the training task that can…
Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization
School of Electrical Engineering and Computer Science, The Pennsylvania State University（宾夕法尼亚州立大学电气工程与计算机科学学院）；Department of Electrical and Computer Engineering, Bucknell University（巴克内尔大学电气与计算机工程系）
SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning
Georgia Institute of Technology（佐治亚理工学院）；TReNDS Center（TReNDS中心）；Google DeepMind, Mila - Quebec AI Institute（谷歌DeepMind、魁北克AI研究所）；Georgia State University（佐治亚州立大学）；Emory University（埃默里大学）；Center for Translational Research in Neuroimaging & Data Science (TReNDS)（神经影像与数据科学转化研究中心（TReNDS））
Tsinghua University（清华大学）；BNRist（北京国家研究院（BNRist））；Key Laboratory of Pervasive Computing, Ministry of Education（教育部普适计算重点实验室）
Federated learning (FL) refers to the learning paradigm that trains machine learning models directly in the decentralized systems consisting of smart edge devices without transmitting the raw data, which avoids the heavy communication costs and privacy concerns. Given the typical heterogeneous data distributions in such situations, the popular FL algorithm \emph{Federated Averaging} (FedAvg) suffers from weight divergence and thus cannot achieve a competitive performance for the global model (denoted as the \emph{initial performance} in FL) compared to centralized methods. In this paper, we pr…
We address the problem of federated learning (FL) where users are distributed and partitioned into clusters. This setup captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task), they can leverage the strength in numbers in order to perform more efficient federated learning. For this new framework of clustered federated learning, we propose the Iterative Federated Clustering Algorithm (IFCA), which alternately estimates the cluster identities of the users and optimizes model pa…
Improving Learning of New Diseases through Knowledge-Enhanced Initialization for Federated Adapter Tuning
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（高性能计算研究所（IHPC），科技研究局（A*STAR））；EVYD Technology（EVYD技术）
Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization
School of Cyber Science and Technology Shenzhen Campus of Sun Yat-sen University（中山大学计算机科学与技术学院深圳校区）；School of Computer Science Faculty of Engineering The University of Sydney（悉尼大学计算机科学学院）；Generative AI Lab College of Computing and Data Science Nanyang Technological University（南洋理工大学计算与数据科学学院生成式人工智能实验室）
The idea of federated learning is to train deep neural network models collaboratively and share them with multiple participants without exposing their private training data to each other. This is highly attractive in the medical domain due to patients' privacy records. However, a recently proposed method called Deep Leakage from Gradients enables attackers to reconstruct data from shared gradients. This study shows how easy it is to reconstruct images for different data initialization schemes and distance measures. We show how data and model architecture influence the optimal choice of initial…
Halıcıoğlu Data Science Institute（哈利西奥卢数据科学研究所）；UC San Diego（加州大学圣地亚哥分校）
In this paper, we address the dichotomy between heterogeneous models and simultaneous training in Federated Learning (FL) via a clustering framework. We define a new clustering model for FL based on the (optimal) local models of the users: two users belong to the same cluster if their local models are close; otherwise they belong to different clusters. A standard algorithm for clustered FL is proposed in \cite{ghosh_efficient_2021}, called \texttt{IFCA}, which requires \emph{suitable} initialization and the knowledge of hyper-parameters like the number of clusters (which is often quite difficu…
Federated learning (FL) is recently surging as a promising decentralized deep learning (DL) framework that enables DL-based approaches trained collaboratively across clients without sharing private data. However, in the context of the central party being active and dishonest, the data of individual clients might be perfectly reconstructed, leading to the high possibility of sensitive information being leaked. Moreover, FL also suffers from the nonindependent and identically distributed (non-IID) data among clients, resulting in the degradation in the inference performance on local clients' dat…
Northeastern University（东北大学）；The Fuqua School of Business（福库商学院）；Duke University（杜克大学）；Center for Statistical Science（统计科学中心）；Tsinghua University（清华大学）
This paper studies the problem of model training under Federated Learning when clients exhibit cluster structure. We contextualize this problem in mixed regression, where each client has limited local data generated from one of $k$ unknown regression models. We design an algorithm that achieves global convergence from any initialization, and works even when local data volume is highly unbalanced -- there could exist clients that contain $O(1)$ data points only. Our algorithm first runs moment descent on a few anchor clients (each with $\tildeΩ(k)$ data points) to obtain coarse model estimates.…
Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
Institute of Engineering Research, Korea University（韩国大学工程研究所）；Kim Jaechul Graduate School of AI, KAIST（韩国科学技术院金在哲人工智能研究生院）；Industrial Management Engineering, Korea University（韩国大学产业管理工程学院）
Federated Distillation (FD) is a popular novel algorithmic paradigm for Federated Learning, which achieves training performance competitive to prior parameter averaging based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work we propose FedAUX, an extension to FD, which, under the same set of assumptions, drastically improves performance by deriving maximum utility from the unlabeled auxiliary data. FedAUX modifies the FD training procedure in two …
Local K-Similarity Constraint for Federated Learning with Label Noise
NAAMII, Nepal Applied Mathematics and Informatics Institute for Research（NAAMII尼泊尔应用数学与信息学研究所）；Rochester Institute of Technology（罗切斯特技术研究所）；West Virginia University（西弗吉尼亚大学）；University of Lausanne（洛桑大学）；Ulsan National Institute of Science & Technology（釜山国家科学与技术研究所）；University of Aberdeen（阿伯丁大学）
Federated learning (FL) enables on-device training over distributed networks consisting of a massive amount of modern smart devices, such as smartphones and IoT (Internet of Things) devices. However, the leading optimization algorithm in such settings, i.e., federated averaging (FedAvg), suffers from heavy communication costs and the inevitable performance drop, especially when the local data is distributed in a non-IID way. To alleviate this problem, we propose two potential solutions by introducing additional mechanisms to the on-device training. The first (FedMMD) is adopting a two-stream m…
DecHW: Heterogeneous Decentralized Federated Learning Exploiting Second-Order Information
School of Information Technology, Deakin University（德肯大学信息科技学院）；Institute for Informatics and Telematics (IIT), Italian National Research Council (CNR)（信息与电信学研究院（IIT），意大利国家研究委员会（CNR））
Federated Learning (FL) is an increasingly popular machine learning paradigm in which multiple nodes try to collaboratively learn under privacy, communication and multiple heterogeneity constraints. A persistent problem in federated learning is that it is not clear what the optimization objective should be: the standard average risk minimization of supervised learning is inadequate in handling several major constraints specific to federated learning, such as communication adaptivity and personalization control. We identify several key desiderata in frameworks for federated learning and introdu…
Towards One-shot Federated Learning: Advances, Challenges, and Future Directions
Department of Electrical Engineering and Information Technology, University of Naples Federico II（那不勒斯费德里科二世大学电气工程与信息科技系）；Department of Mathematics and Applications "R. Caccioppoli", University of Naples Federico II（那不勒斯费德里科二世大学数学与应用系）；Department of Mathematics, Indian Institute of Technology Indore（印度理工学院Indore分校数学系）
Onboarding Without Forgetting: Hypernetwork Personalization with Data-Free Replay for Personalized Federated Learning
VinUni-Illinois Smart Health Center（VinUni-Illinois智能健康中心）；College of Engineering & Computer Science（工程与计算机科学学院）；University of Notre Dame（诺特丹大学）；Technische Universität Berlin（柏林技术大学）
OledFL: Unleashing the Potential of Decentralized Federated Learning via Opposite Lookahead Enhancement
National University of Defense Technology（国防科技大学）；Hebei University of Technology（河北工业大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区网络空间科学与技术学院）
On the Limits of Momentum in Decentralized and Federated Optimization
Department of Computer and Control Engineering, Polytechnic of Turin（计算机与控制工程系，都灵理工学院）；Thomas Lord Department of Computer Science, USC Viterbi School of Engineering（托马斯·劳德计算机科学系，USC维特里商学院）
Center for Frontier AI Research（前沿人工智能研究中心）；A*STAR（新加坡科技研究局）；Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）
Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model. However, this would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous. As a result, current federated learning methods often require a large number of communication rounds in training. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning. In this work, we first introduce a meta knowledge representati…
Distilling A Universal Expert from Clustered Federated Learning
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education（教育部符号计算与知识工程重点实验室）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney（悉尼大学技术学院澳大利亚人工智能研究所）；College of Computer Science, Inner Mongolia University（内蒙古大学计算机科学学院）
Hong Kong University of Science and Technology（香港科技大学）；University of Science and Technology China（中国科学技术大学）；University of Cambridge（剑桥大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
Federated learning alleviates the privacy risk in distributed learning by transmitting only the local model updates to the central server. However, it faces challenges including statistical heterogeneity of clients' datasets and resource constraints of client devices, which severely impact the training performance and user experience. Prior works have tackled these challenges by combining personalization with model compression schemes including quantization and pruning. However, the pruning is data-dependent and thus must be done on the client side which requires considerable computation cost.…
Efficient Wireless Federated Learning via Low-Rank Gradient Factorization
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）
