The Web is naturally heterogeneous with user devices, geographic regions, browsing patterns, and contexts all leading to highly diverse, unique datasets. Federated Learning (FL) is an important paradigm for the Web because it enables privacy-preserving, collaborative machine learning across diverse user devices, web services and clients without needing to centralize sensitive data. However, its performance degrades severely under non-IID client distributions that is prevalent in real-world web systems. In this work, we propose a new training paradigm - Iterative Federated Adaptation (IFA) - that enhances generalization in heterogeneous federated settings through generation-wise forget and evolve strategy. Specifically, we divide training into multiple generations and, at the end of each, select a fraction of model parameters (a) randomly or (b) from the later layers of the model and reinitialize them. This iterative forget and evolve schedule allows the model to escape local minima and preserve globally relevant representations. Extensive experiments on CIFAR-10, MIT-Indoors, and Stanford Dogs datasets show that the proposed approach improves global accuracy, especially when the data cross clients are Non-IID. This method can be implemented on top any federated algorithm to improve its generalization performance. We observe an average of 21.5%improvement across datasets. This work advances the vision of scalable, privacy-preserving intelligence for real-world heterogeneous and distributed web systems.
Class-wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory Generation
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）；College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Cyberspace, Hangzhou Dianzi University（杭州电子科技大学网络空间学院）
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Northwestern University（西北大学）；University of Technology Sydney（悉尼科技大学）
Federated learning (FL) has attracted growing attention via data-private collaborative training on decentralized clients. However, most existing methods unrealistically assume object classes of the overall framework are fixed over time. It makes the global model suffer from significant catastrophic forgetting on old classes in real-world scenarios, where local clients often collect new classes continuously and have very limited storage memory to store old classes. Moreover, new clients with unseen new classes may participate in the FL training, further aggravating the catastrophic forgetting o…
We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the …
$f$-FUM: Federated Unlearning via min--max and $f$-divergence
Department of Statistics, Universite de Geneve, Geneva, Switzerland（日内瓦大学统计系）；University of Cambridge, Cambridge, United Kingdom（剑桥大学）；The Alan Turing Institute, London, UK（艾伦·图灵研究所）
Onboarding Without Forgetting: Hypernetwork Personalization with Data-Free Replay for Personalized Federated Learning
VinUni-Illinois Smart Health Center（VinUni-Illinois智能健康中心）；College of Engineering & Computer Science（工程与计算机科学学院）；University of Notre Dame（诺特丹大学）；Technische Universität Berlin（柏林技术大学）
Federated Learning (FL) has emerged as a de facto machine learning area and received rapid increasing research interests from the community. However, catastrophic forgetting caused by data heterogeneity and partial participation poses distinctive challenges for FL, which are detrimental to the performance. To tackle the problems, we propose a new FL approach (namely GradMA), which takes inspiration from continual learning to simultaneously correct the server-side and worker-side update directions as well as take full advantage of server's rich computing and memory resources. Furthermore, we el…
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that …
ToFU: Transforming How Federated Learning Systems Forget User Data
School of Computer Science（计算机科学学院）；Statistics, Trinity College Dublin, Ireland（统计学，都柏林信任学院，爱尔兰）；School of Computer Science, University College Dublin, Ireland（计算机科学学院，都柏林大学学院，爱尔兰）
Overcoming Catastrophic Forgetting in Federated Class-Incremental Learning via Federated Global Twin Generator
College of Engineering & Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；Vietnam National University - Ho Chi Minh city, University of Science（越南国家大学胡志明市分校自然科学大学）；Technische Universität Berlin（柏林工业大学）
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on a central server, thereby producing an effective global model. Although a variety of FL algorithms have been proposed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients. We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during…
Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Arizona State University（亚利桑那州立大学）；University of California San Diego（加州大学圣地亚哥分校）；Purdue University（普渡大学）；University of Maryland College Park（马里兰大学学院市分校）
FedRef: Bayesian Fine-Tuning using a Reference Model to Mitigate Catastrophic Forgetting for Heterogeneous Federated Learning
School of Computer Science and Engineering（计算机科学与工程学院）；Center for Biosystems and Biotech Data Science（生物系统与生物技术数据科学中心）；Ghent University Global Campus, Incheon, Republic of Korea（因斯布鲁克大学全球校区，韩国）
Federated Learning has been introduced as a new machine learning paradigm enhancing the use of local devices. At a server level, FL regularly aggregates models learned locally on distributed clients to obtain a more general model. In this way, no private data is sent over the network, and the communication cost is reduced. However, current solutions rely on the availability of large amounts of stored data at the client side in order to fine-tune the models sent by the server. Such setting is not realistic in mobile pervasive computing where data storage must be kept low and data characteristic…
Federated Class-Incremental Learning with New-Class Augmented Self-Distillation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Laboratory（中关村实验室）；Beijing Jiaotong University（北京交通大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；UC Santa Cruz（加州大学圣克鲁兹分校）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigoro…
FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；National Engineering Research Center of Disaster Backup and Recovery, Beijing University of Posts and Telecommunications（北京邮电大学国家灾备工程技术研究中心）；JIUTIAN Research, China Mobile（中国移动九天研究院）；School of Cyberspace Security, Beijing University of Posts and Telecommunications（北京邮电大学网络空间安全学院）；University of Leicester（莱斯特大学）
The heterogeneity of hardware and data is a well-known and studied problem in the community of Federated Learning (FL) as running under heterogeneous settings. Recently, custom-size client models trained with Knowledge Distillation (KD) has emerged as a viable strategy for tackling the heterogeneity challenge. However, previous efforts in this direction are aimed at client model tuning rather than their impact onto the knowledge aggregation of the global model. Despite performance of global models being the primary objective of FL systems, under heterogeneous settings client models have receiv…
Federated learning~(FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints. Existing iterative model averaging based FL algorithms require a large number of communication rounds to obtain a well-performed model due to extremely unbalanced and non-i.i.d data partitioning among different clients. Thus, we propose FedDM to build the global training objective from multiple local surrogate functions, which enables the server to gain a more global view of the loss landscape. In…
FedCARE: Federated Unlearning with Conflict-Aware Projection and Relearning-Resistant Recovery
School of Cyber Engineering, Xidian University（电子科技大学信息工程学院）；RMIT University（皇家墨尔本理工大学）；Commonwealth Scientific and Industrial Research Organisation (CSIRO)（澳大利亚联邦科学与工业研究组织）；UNSW Sydney（新南威尔士大学悉尼分校）；University of California, Davis（加州大学戴维斯分校）
Xiamen University（厦门大学）；University of Melbourne（墨尔本大学）
Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the …
School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；Department of Computer Science, Purdue University（普渡大学计算机科学系）
Federated learning enables a global machine learning model to be trained collaboratively by distributed, mutually non-trusting learning agents who desire to maintain the privacy of their training data and their hardware. A global model is distributed to clients, who perform training, and submit their newly-trained model to be aggregated into a superior model. However, federated learning systems are vulnerable to interference from malicious learning agents who may desire to prevent training or induce targeted misclassification in the resulting global model. A class of Byzantine-tolerant aggrega…
