Malicious clients can attack federated learning systems using malicious data, including backdoor samples, during the training phase. The compromised global model will perform well on the validation dataset designed for the task, but a small subset of data with backdoor patterns may trigger the model to make a wrong prediction. There has been an arms race between attackers who tried to conceal attacks and defenders who tried to detect attacks during the aggregation stage of training on the server-side. In this work, we propose a new and effective method to mitigate backdoor attacks after the training phase. Specifically, we design a federated pruning method to remove redundant neurons in the network and then adjust the model's extreme weight values. Our experiments conducted on distributed Fashion-MNIST show that our method can reduce the average attack success rate from 99.7% to 1.9% with a 5.5% loss of test accuracy on the validation dataset. To minimize the pruning influence on test accuracy, we can fine-tune after pruning, and the attack success rate drops to 6.4%, with only a 1.7% loss of test accuracy. Further experiments under Distributed Backdoor Attacks on CIFAR-10 also show promising results that the average attack success rate drops more than 70% with less than 2% loss of test accuracy on the validation dataset.
Despite the potential of federated learning, it is known to be vulnerable to backdoor attacks. Many robust federated aggregation methods are proposed to reduce the potential backdoor risk. However, they are mainly validated in the CV field. In this paper, we find that NLP backdoors are hard to defend against than CV, and we provide a theoretical analysis that the malicious update detection error probabilities are determined by the relative backdoor strengths. NLP attacks tend to have small relative backdoor strengths, which may result in the failure of robust federated aggregation methods for …
Huazhong University of Science and Technology（华中科技大学）
Edge devices in federated learning usually have much more limited computation and communication resources compared to servers in a data center. Recently, advanced model compression methods, like the Lottery Ticket Hypothesis, have already been implemented on federated learning to reduce the model size and communication cost. However, Backdoor Attack can compromise its implementation in the federated learning scenario. The malicious edge device trains the client model with poisoned private data and uploads parameters to the center, embedding a backdoor to the global shared model after unwitting…
Federated learning enables thousands of participants to construct a deep learning model without sharing their private training data with each other. For example, multiple smartphones can jointly train a next-word predictor for keyboards without revealing what individual users type. We demonstrate that any participant in federated learning can introduce hidden backdoor functionality into the joint global model, e.g., to ensure that an image classifier assigns an attacker-chosen label to images with certain features, or that a word predictor completes certain sentences with an attacker-chosen wo…
Tongji University（同济大学）；Pennsylvania State University（宾夕法尼亚州立大学）
Federated Learning (FL) is a popular distributed machine learning paradigm that enables jointly training a global model without sharing clients' data. However, its repetitive server-client communication gives room for backdoor attacks with aim to mislead the global model into a targeted misprediction when a specific trigger pattern is presented. In response to such backdoor threats on federated learning, various defense measures have been proposed. In this paper, we study whether the current defense mechanisms truly neutralize the backdoor threats from federated learning in a practical setting…
The goal of federated learning (FL) is to train one global model by aggregating model parameters updated independently on edge devices without accessing users' private data. However, FL is susceptible to backdoor attacks where a small fraction of malicious agents inject a targeted misclassification behavior in the global model by uploading polluted model updates to the server. In this work, we propose DifFense, an automated defense framework to protect an FL system from backdoor attacks by leveraging differential testing and two-step MAD outlier detection, without requiring any previous knowle…
SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning
Department of Information and Communication Engineering, Major in Information Security, Chosun University（信息与通信工程系，信息安全专业，全州大学）；AI Data Security Research Section, Electronics and Telecommunications Research Institute（人工智能数据安全研究部，电子电信研究院）；Department of Convergence Science, Kongju National University（融合科学系， Kongju国立大学）
University of Maryland（马里兰大学）；Google（谷歌）；Johns Hopkins University（约翰斯·霍普金斯大学）；New York University（纽约大学）
Federated learning is particularly susceptible to model poisoning and backdoor attacks because individual users have direct control over the training data and model updates. At the same time, the attack power of an individual user is limited because their updates are quickly drowned out by those of many other users. Existing attacks do not account for future behaviors of other users, and thus require many sequential updates and their effects are quickly erased. We propose an attack that anticipates and accounts for the entire federated learning pipeline, including behaviors of other clients, a…
Recent studies have shown that federated learning (FL) is vulnerable to poisoning attacks that inject a backdoor into the global model. These attacks are effective even when performed by a single client, and undetectable by most existing defensive techniques. In this paper, we propose Backdoor detection via Feedback-based Federated Learning (BAFFLE), a novel defense to secure FL against backdoor attacks. The core idea behind BAFFLE is to leverage data of multiple clients not only for training but also for uncovering model poisoning. We exploit the availability of diverse datasets at the variou…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Zhejiang University（浙江大学）；IBM Research（IBM研究院）
Federated Learning (FL) as a distributed learning paradigm that aggregates information from diverse clients to train a shared global model, has demonstrated great success. However, malicious clients can perform poisoning attacks and model replacement to introduce backdoors into the trained global model. Although there have been intensive studies designing robust aggregation methods and empirical robust federated training protocols against backdoors, existing approaches lack robustness certification. This paper provides the first general framework, Certifiably Robust Federated Learning (CRFL), …
FedBAP: Backdoor Defense via Benign Adversarial Perturbation in Federated Learning
School of Cyber Science and Engineering, Wuhan University（武汉大学计算机科学与工程学院）；Department of Computer and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机与人工智能学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）
Federated Learning (FL) allows multiple clients to collaboratively train a Neural Network (NN) model on their private data without revealing the data. Recently, several targeted poisoning attacks against FL have been introduced. These attacks inject a backdoor into the resulting model that allows adversary-controlled inputs to be misclassified. Existing countermeasures against backdoor attacks are inefficient and often merely aim to exclude deviating models from the aggregation. However, this approach also removes benign models of clients with deviating data distributions, causing the aggregat…
Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
Backdoor attacks are dangerous and difficult to prevent in federated learning (FL), where training data is sourced from untrusted clients over long periods of time. These difficulties arise because: (a) defenders in FL do not have access to raw training data, and (b) a new phenomenon we identify called backdoor leakage causes models trained continuously to eventually suffer from backdoors due to cumulative errors in defense mechanisms. We propose shadow learning, a framework for defending against backdoor attacks in the FL setting under long-range training. Shadow learning trains two models in…
BoBa: Boosting Backdoor Detection through Data Distribution Inference in Federated Learning
University of South Florida（南佛罗里达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；University of Kentucky（肯塔基大学）
Is the Trigger Essential? A Feature-Based Triggerless Backdoor Attack in Vertical Federated Learning
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术重点实验室（北京大学））；School of Computer Science, Peking University（北京大学计算机学院）；Zhongguancun Laboratory（中关村实验室）
The decentralized nature of federated learning makes detecting and defending against adversarial attacks a challenging task. This paper focuses on backdoor attacks in the federated learning setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining good performance on the main task. Unlike existing works, we allow non-malicious clients to have correctly labeled samples from the targeted tasks. We conduct a comprehensive study of backdoor attacks and defenses for the EMNIST dataset, a real-life, user-partitioned, and non-iid dataset. W…
Southeast University（东南大学）；Princeton University（普林斯顿大学）；University of California at Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）
Due to their decentralized nature, federated learning (FL) systems have an inherent vulnerability during their training to adversarial backdoor attacks. In this type of attack, the goal of the attacker is to use poisoned updates to implant so-called backdoors into the learned model such that, at test time, the model's outputs can be fixed to a given target for certain inputs. (As a simple toy example, if a user types "people from New York" into a mobile keyboard app that uses a backdoored next word prediction model, then the model could autocomplete the sentence to "people from New York are ru…
FilterFL: Knowledge Filtering-based Data-Free Backdoor Defense for Federated Learning
East China Normal University, Shanghai, China（华东师范大学）；Singapore Management University, Singapore（新加坡管理学院）；Nanyang Technological University, Singapore（南洋理工大学）；National University of Singapore, Singapore（国立新加坡大学）
Cooperative Decentralized Backdoor Attacks on Vertical Federated Learning
Elmore Family School of Electrical and Computer Engineering, Purdue University（电子工程学院，普渡大学）；Department of Electrical and Computer Engineering, University of Akron（电气与计算机工程系，阿克隆大学）；Department of Electrical Engineering, University at Buffalo-SUNY（电气工程系，布法罗大学）
Federated Learning (FL) is a new machine learning framework, which enables millions of participants to collaboratively train machine learning model without compromising data privacy and security. Due to the independence and confidentiality of each client, FL does not guarantee that all clients are honest by design, which makes it vulnerable to adversarial attack naturally. In this paper, we focus on dynamic backdoor attacks under FL setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining a good performance on the main task, current…
FedSurrogate: Backdoor Defense in Federated Learning via Layer Criticality and Surrogate Replacement
The University of Manchester, UK（英国曼彻斯特大学）；Institute for Infocomm Research, A*STAR, Singapore（新加坡信息通信研究所）；Monash University, Australia（澳大利亚墨尔本大学）；COSIC, KU Leuven, Belgium（比利时列日大学COSIC实验室）
Beyond Traditional Threats: A Persistent Backdoor Attack on Federated Learning
College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）；Southampton Ocean Engineering Joint Institute, Harbin Engineering University（哈尔滨工程大学南安普顿海洋工程联合学院）
Inserting a backdoor into the joint model in federated learning (FL) is a recent threat raising concerns. Existing studies mostly focus on developing effective countermeasures against this threat, assuming that backdoored local models, if any, somehow reveal themselves by anomalies in their gradients. However, this assumption needs to be elaborated by identifying specifically which gradients are more likely to indicate an anomaly to what extent under which conditions. This is an important issue given that neural network models usually have huge parametric space and consist of a large number of…
Enhancing the Effectiveness and Durability of Backdoor Attacks in Federated Learning through Maximizing Task Distinction
School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Trustworthy and General Artificial Intelligence Laboratory, School of Engineering, Westlake University（西湖大学工程学院可信与通用人工智能实验室）
VinUniversity（文大大学）；Hanoi University of Science and Technology（河内科技大学）
Federated learning (FL) is a machine learning (ML) approach that allows the use of distributed data without compromising personal privacy. However, the heterogeneous distribution of data among clients in FL can make it difficult for the orchestration server to validate the integrity of local model updates, making FL vulnerable to various threats, including backdoor attacks. Backdoor attacks involve the insertion of malicious functionality into a targeted model through poisoned updates from malicious clients. These attacks can cause the global model to misbehave on specific inputs while appeari…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the performance of the trained model on specific sub-tasks (e.g., by classifying green cars as frogs). A range of FL backdoor attacks have been introduced in the literature, but also methods to defend against them, and it is currently an open question whether FL systems can be tailored to be robust against backdoors. In this work, we provide evidence to the contrary. We first establish that, in the general case, robustness t…
Exploiting Layer-Specific Vulnerabilities to Backdoor Attack in Federated Learning
School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran（塔里斯坦大学电气与计算机工程学院）；School of Electrical and Computer Engineering, Shiraz University, Shiraz, Iran（谢尔兹大学电气与计算机工程学院）；Okinawa Institute of Science and Technology, Okinawa, Japan（冲绳科学技术研究院）；School of Computer Science, Institute for Research in Fundamental Sciences (IPM), Tehran, Iran（基础科学研究院（IPM）计算机科学学院）
The Hong Kong University of Science and Technology（香港科技大学）
Federated learning is vulnerable to various attacks, such as model poisoning and backdoor attacks, even if some existing defense strategies are used. To address this challenge, we propose an attack-adaptive aggregation strategy to defend against various attacks for robust federated learning. The proposed approach is based on training a neural network with an attention mechanism that learns the vulnerability of federated learning models from a set of plausible attacks. To the best of our knowledge, our aggregation strategy is the first one that can be adapted to defend against various attacks i…
