Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Alibaba Group（阿里巴巴集团）；Shandong University（山东大学）；Joint SDU-NTU Centre for Artificial Intelligence Research（山东大学-南洋理工大学人工智能联合研究院）；Hong Kong University of Science and Technology（香港科技大学）；WeBank（微众银行）
In parallel with the rapid adoption of Artificial Intelligence (AI) empowered by advances in AI research, there have been growing awareness and concerns of data privacy. Recent significant developments in the data regulation landscape have prompted a seismic shift in interest towards privacy-preserving AI. This has contributed to the popularity of Federated Learning (FL), the leading paradigm for the training of machine learning models on data silos in a privacy-preserving manner. In this survey, we explore the domain of Personalized FL (PFL) to address the fundamental challenges of FL on heterogeneous data, a universal characteristic inherent in all real-world datasets. We analyze the key motivations for PFL and present a unique taxonomy of PFL techniques categorized according to the key challenges and personalization strategies in PFL. We highlight their key ideas, challenges and opportunities and envision promising future trajectories of research towards new PFL architectural design, realistic PFL benchmarking, and trustworthy PFL approaches.
PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmark
Shanghai Jiao Tong University（上海交通大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Queen’s University Belfast（贝尔法斯特女王大学）；Stevens Institute of Technology（史蒂文斯理工学院）
South China University of Technology（华南理工大学）；Eindhoven University of Technology（埃因霍温理工大学）；JD Explore Academy（京东探索研究院）
Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user. To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user. However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory. In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme…
Personalized Federated Learning (pFL), which utilizes and deploys distinct local models, has gained increasing attention in recent years due to its success in handling the statistical heterogeneity of FL clients. However, standardized evaluation and systematical analysis of diverse pFL methods remain a challenge. Firstly, the highly varied datasets, FL simulation settings and pFL implementations prevent easy and fair comparisons of pFL methods. Secondly, the current pFL literature diverges in the adopted evaluation and ablation protocols. Finally, the effectiveness and robustness of pFL method…
Federated learning (FL for simplification) is a distributed machine learning technique that utilizes global servers and collaborative clients to achieve privacy-preserving global model training without direct data sharing. However, heterogeneous data problem, as one of FL's main problems, makes it difficult for the global model to perform effectively on each client's local data. Thus, personalized federated learning (PFL for simplification) aims to improve the performance of the model on local data as much as possible. Bayesian learning, where the parameters of the model are seen as random var…
MH-pFLGB: Model Heterogeneous personalized Federated Learning via Global Bypass for Medical Image Analysis
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）；State University of New York at Buffalo（纽约州立大学布法罗分校）
Singapore University of Design and Technology（新加坡科技设计大学）；East China Normal University（华东师范大学）；Zhejiang University（浙江大学）
Personalized Federated Learning (PFL) is a new Federated Learning (FL) paradigm, particularly tackling the heterogeneity issues brought by various mobile user equipments (UEs) in mobile edge computing (MEC) networks. However, due to the ever-increasing number of UEs and the complicated administrative work it brings, it is desirable to switch the PFL algorithm from its conventional two-layer framework to a multiple-layer one. In this paper, we propose hierarchical PFL (HPFL), an algorithm for deploying PFL over massive MEC networks. The UEs in HPFL are divided into multiple clusters, and the UE…
MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysis
School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）；State University of New York at Buffalo（纽约州立大学布法罗分校）
The Hong Kong Polytechnic University（香港理工大学）；HUST（华中科技大学）
In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we form…
Personalized Federated Learning (PFL) aims to learn personalized models for each client based on the knowledge across all clients in a privacy-preserving manner. Existing PFL methods generally assume that the underlying global data across all clients are uniformly distributed without considering the long-tail distribution. The joint problem of data heterogeneity and long-tail distribution in the FL environment is more challenging and severely affects the performance of personalized models. In this paper, we propose a PFL method called Federated Learning with Adversarial Feature Augmentation (F…
FedAH: Aggregated Head for Personalized Federated Learning
College of Computer Science Chongqing University Chongqing, China；School of Computer Science；Technology Beijing Institute of Technology Beijing, China；Technology Xi'an Jiaotong University Xi'an, China
Clust-PSI-PFL: A Population Stability Index Approach for Clustered Non-IID Personalized Federated Learning
Department of Computer, Control；Management Engineering Sapienza University of Rome Rome, Italy；Telef\'onica Research Barcelona, Spain；Keysight AI Labs Barcelona, Spain
Bar Ilan University（巴伊兰大学）；NVIDIA Research（英伟达研究院）
In Federated Learning (FL), multiple clients collaborate to learn a shared model through a central server while keeping data decentralized. Personalized Federated Learning (PFL) further extends FL by learning a personalized model per client. In both FL and PFL, all clients participate in the training process and their labeled data are used for training. However, in reality, novel clients may wish to join a prediction service after it has been deployed, obtaining predictions for their own \textbf{unlabeled} data. Here, we introduce a new learning setup, On-Demand Unlabeled PFL (OD-PFL), where a…
The Hong Kong Polytechnic University（香港理工大学）；Beijing Institute of Technology（北京理工大学）
Facing the challenge of statistical diversity in client local data distribution, personalized federated learning (PFL) has become a growing research hotspot. Although the state-of-the-art methods with model similarity-based pairwise collaboration have achieved promising performance, they neglect the fact that model aggregation is essentially a collaboration process within the coalition, where the complex multiwise influences take place among clients. In this paper, we first apply Shapley value (SV) from coalition game theory into the PFL scenario. To measure the multiwise collaboration among a…
FedAPA: Server-side Gradient-Based Adaptive Personalized Aggregation for Federated Learning on Heterogeneous Data
College of Information Science and Technology, Jinan University（暨南大学信息科学技术学院）；College of Information and Computational Science, Jilin University（吉林大学信息与计算科学学院）；University of Toronto（多伦多大学）；The Hong Kong Polytechnic University（香港理工大学）
National University of Defense Technology（国防科技大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to…
Towards Layer-Wise Personalized Federated Learning: Adaptive Layer Disentanglement via Conflicting Gradients
Pusan National University（釜山国立大学）；Hanoi University of Technology（河内工业大学）；University of Sydney（悉尼大学）；Trinity College of Dublin（都柏林圣三一学院）；Zhejiang University（浙江大学）
Singapore University of Design and Technology（新加坡科技设计大学）；Shenzhen University（深圳大学）；East China Normal University（华东师范大学）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Personalized Federated Learning (PFL) is a new Federated Learning (FL) approach to address the heterogeneity issue of the datasets generated by distributed user equipments (UEs). However, most existing PFL implementations rely on synchronous training to ensure good convergence performances, which may lead to a serious straggler problem, where the training time is heavily prolonged by the slowest UE. To address this issue, we propose a semi-synchronous PFL algorithm, termed as Semi-Synchronous Personalized FederatedAveraging (PerFedS$^2$), over mobile edge networks. By jointly optimizing the wi…
The Pennsylvania State University（宾夕法尼亚州立大学）；NEC Laboratories America（美国NEC实验室）；Florida International University（佛罗里达国际大学）；AWS AI Labs, Amazon（亚马逊AWS人工智能实验室）
Personalized Federated Learning (PFL) which collaboratively trains a federated model while considering local clients under privacy constraints has attracted much attention. Despite its popularity, it has been observed that existing PFL approaches result in sub-optimal solutions when the joint distribution among local clients diverges. To address this issue, we present Federated Modular Network (FedMN), a novel PFL approach that adaptively selects sub-modules from a module pool to assemble heterogeneous neural architectures for different clients. FedMN adopts a light-weighted routing hypernetwo…
Convergence-Privacy-Fairness Trade-Off in Personalized Federated Learning
Beijing University of Posts and Telecommunications（北京邮电大学）；Macquarie University（麦考瑞大学）；Peng Cheng Laboratory（鹏城实验室）；University of Technology Sydney（悉尼大学）；University of Manitoba（曼尼托巴大学）
Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）；University of Washington（华盛顿大学）；University of Maryland, College Park（马里兰大学帕克分校）
Knowledge sharing and model personalization are two key components in the conceptual framework of personalized federated learning (PFL). Existing PFL methods focus on proposing new model personalization mechanisms while simply implementing knowledge sharing by aggregating models from all clients, regardless of their relation graph. This paper aims to enhance the knowledge-sharing process in PFL by leveraging the graph-based structural information among clients. We propose a novel structured federated learning (SFL) framework to learn both the global and personalized models simultaneously using…
Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Bilkent University（比尔肯特大学）；University of Southern California（南加州大学）
Multi-institutional collaborations are key for learning generalizable MRI synthesis models that translate source- onto target-contrast images. To facilitate collaboration, federated learning (FL) adopts decentralized training and mitigates privacy concerns by avoiding sharing of imaging data. However, FL-trained synthesis models can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident when common or variable translation tasks are prescribed across sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) to improve reliabil…
