State Key Laboratory of Integrated Services Networks (ISN)（综合业务网理论及关键技术国家重点实验室）；Shaanxi Key Laboratory of Network and System Security, Xidian University（西安电子科技大学陕西省网络与系统安全重点实验室）；School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；Ant Group（蚂蚁集团）；College of Computer and Data Science, Fuzhou University（福州大学计算机与数据科学学院）
Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, Jianfeng Ma
Backdoor injection attack is an emerging threat to the security of neural networks, however, there still exist limited effective defense methods against the attack. In this paper, we propose BAERASE, a novel method that can erase the backdoor injected into the victim model through machine unlearning. Specifically, BAERASE mainly implements backdoor defense in two key steps. First, trigger pattern recovery is conducted to extract the trigger patterns infected by the victim model. Here, the trigger pattern recovery problem is equivalent to the one of extracting an unknown noise distribution from the victim model, which can be easily resolved by the entropy maximization based generative model. Subsequently, BAERASE leverages these recovered trigger patterns to reverse the backdoor injection procedure and induce the victim model to erase the polluted memories through a newly designed gradient ascent based machine unlearning method. Compared with the previous machine unlearning solutions, the proposed approach gets rid of the reliance on the full access to training data for retraining and shows higher effectiveness on backdoor erasing than existing fine-tuning or pruning methods. Moreover, experiments show that BAERASE can averagely lower the attack success rates of three kinds of state-of-the-art backdoor attacks by 99\% on four benchmark datasets.
The goal of federated learning (FL) is to train one global model by aggregating model parameters updated independently on edge devices without accessing users' private data. However, FL is susceptible to backdoor attacks where a small fraction of malicious agents inject a targeted misclassification behavior in the global model by uploading polluted model updates to the server. In this work, we propose DifFense, an automated defense framework to protect an FL system from backdoor attacks by leveraging differential testing and two-step MAD outlier detection, without requiring any previous knowle…
FedBAP: Backdoor Defense via Benign Adversarial Perturbation in Federated Learning
School of Cyber Science and Engineering, Wuhan University（武汉大学计算机科学与工程学院）；Department of Computer and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机与人工智能学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）
Tongji University（同济大学）；Pennsylvania State University（宾夕法尼亚州立大学）
Federated Learning (FL) is a popular distributed machine learning paradigm that enables jointly training a global model without sharing clients' data. However, its repetitive server-client communication gives room for backdoor attacks with aim to mislead the global model into a targeted misprediction when a specific trigger pattern is presented. In response to such backdoor threats on federated learning, various defense measures have been proposed. In this paper, we study whether the current defense mechanisms truly neutralize the backdoor threats from federated learning in a practical setting…
Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
Backdoor attacks are dangerous and difficult to prevent in federated learning (FL), where training data is sourced from untrusted clients over long periods of time. These difficulties arise because: (a) defenders in FL do not have access to raw training data, and (b) a new phenomenon we identify called backdoor leakage causes models trained continuously to eventually suffer from backdoors due to cumulative errors in defense mechanisms. We propose shadow learning, a framework for defending against backdoor attacks in the FL setting under long-range training. Shadow learning trains two models in…
Hypnopaedia-Aware Machine Unlearning via Psychometrics of Artificial Mental Imagery
National Institute of Informatics（国家信息研究所）；Feng Chia University（逢甲大学）；Lancaster University（兰卡斯特大学）；University of Melbourne（墨尔本大学）；Graduate University for Advanced Studies（高级研究大学）；University of Tokyo（东京大学）
VinUniversity（文大大学）；Hanoi University of Science and Technology（河内科技大学）
Federated learning (FL) is a machine learning (ML) approach that allows the use of distributed data without compromising personal privacy. However, the heterogeneous distribution of data among clients in FL can make it difficult for the orchestration server to validate the integrity of local model updates, making FL vulnerable to various threats, including backdoor attacks. Backdoor attacks involve the insertion of malicious functionality into a targeted model through poisoned updates from malicious clients. These attacks can cause the global model to misbehave on specific inputs while appeari…
Defending the Edge: Representative-Attention Defense against Backdoor Attacks in Federated Learning
Dept. of Computer Science Faculty of Engineering University of Manchester Manchester, United Kingdom；Dept. of Computer Science Faculty of Engineering University of Manchester \& MBZUAI Abu Dhabi, United Arab Emirates；Dept. of Computer Science Faculty of Engineering University of Manchester \& KU Leuven Manchester, United Kingdom
Federated Learning (FL) is a distributed learning paradigm that enables different parties to train a model together for high quality and strong privacy protection. In this scenario, individual participants may get compromised and perform backdoor attacks by poisoning the data (or gradients). Existing work on robust aggregation and certified FL robustness does not study how hardening benign clients can affect the global model (and the malicious clients). In this work, we theoretically analyze the connection among cross-entropy loss, attack success rate, and clean accuracy in this setting. Moreo…
Poisoning Attacks and Defenses to Federated Unlearning
ShanghaiTech University（上海科技大学）；Xidian University（西安电子科技大学）；North Carolina State University（北卡罗来纳州立大学）；University of North Texas（北德克萨斯大学）；University of Louisville（路易斯维尔大学）
ETH Zürich（苏黎世联邦理工学院）；Princeton University（普林斯顿大学）
The right to be forgotten, also known as the right to erasure, is the right of individuals to have their data erased from an entity storing it. The status of this long held notion was legally solidified recently by the General Data Protection Regulation (GDPR) in the European Union. Consequently, there is a need for mechanisms whereby users can verify if service providers comply with their deletion requests. In this work, we take the first step in proposing a formal framework to study the design of such verification mechanisms for data deletion requests -- also known as machine unlearning -- i…
Since there are multiple parties in collaborative learning, malicious parties might manipulate the learning process for their own purposes through backdoor attacks. However, most of existing works only consider the federated learning scenario where data are partitioned by samples. The feature-partitioned learning can be another important scenario since in many real world applications, features are often distributed across different parties. Attacks and defenses in such scenario are especially challenging when the attackers have no labels and the defenders are not able to access the data and mo…
Machine learning models are vulnerable to data-poisoning attacks, in which an attacker maliciously modifies the training set to change the prediction of a learned model. In a trigger-less attack, the attacker can modify the training set but not the test inputs, while in a backdoor attack the attacker can also modify test inputs. Existing model-agnostic defense approaches either cannot handle backdoor attacks or do not provide effective certificates (i.e., a proof of a defense). We present BagFlip, a model-agnostic certified approach that can effectively defend against both trigger-less and bac…
Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Injection, Attack and Erasure: Revocable Backdoor Attacks via Machine Unlearning
Engineering Research Center of Transportation Information and Safety (ERCTIS), MoE of China, School of Computer Science and Artificial Intelligence, Wuhan University of Technology（交通运输信息与安全工程研究中心（ERCTIS）、中国教育部、武汉理工大学计算机科学与人工智能学院）
Fusing Pruned and Backdoored Models: Optimal Transport-based Data-free Backdoor Mitigation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Backdoor attack intends to embed hidden backdoor into deep neural networks (DNNs), such that the attacked model performs well on benign samples, whereas its prediction will be maliciously changed if the hidden backdoor is activated by the attacker defined trigger. This threat could happen when the training process is not fully controlled, such as training on third-party data-sets or adopting third-party models. There has been a lot of research and different methods to defend such type of backdoor attacks, one being robustness-aware perturbation-based defense method. This method mainly exploits…
Virginia Tech（弗吉尼亚理工大学）；University of Michigan（密歇根大学）
We propose a minimax formulation for removing backdoors from a given poisoned model based on a small set of clean data. This formulation encompasses much of prior work on backdoor removal. We propose the Implicit Bacdoor Adversarial Unlearning (I-BAU) algorithm to solve the minimax. Unlike previous work, which breaks down the minimax into separate inner and outer problems, our algorithm utilizes the implicit hypergradient to account for the interdependence between inner and outer optimization. We theoretically analyze its convergence and the generalizability of the robustness gained by solving…
Reconstructive Neuron Pruning for Backdoor Defense
Xidian University（西安电子科技大学）；Fudan University（复旦大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Copenhagen（哥本哈根大学）；Sony AI（索尼人工智能）；University of Illinois at Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）
Backdoor (Trojan) attacks are emerging threats against deep neural networks (DNN). A DNN being attacked will predict to an attacker-desired target class whenever a test sample from any source class is embedded with a backdoor pattern; while correctly classifying clean (attack-free) test samples. Existing backdoor defenses have shown success in detecting whether a DNN is attacked and in reverse-engineering the backdoor pattern in a "post-training" regime: the defender has access to the DNN to be inspected and a small, clean dataset collected independently, but has no access to the (possibly poi…
Pennsylvania State University（宾夕法尼亚州立大学）；Anomalee Inc.（Anomalee 公司）；Imperial College（帝国理工学院）
We demonstrate a backdoor attack on a deep neural network used for regression. The backdoor attack is localized based on training-set data poisoning wherein the mislabeled samples are surrounded by correctly labeled ones. We demonstrate how such localization is necessary for attack success. We also study the performance of a backdoor defense using gradient-based discovery of local error maximizers. Local error maximizers which are associated with significant (interpolation) error, and are proximal to many training samples, are suspicious. This method is also used to accurately train for deep r…
Deep Learning backdoor attacks have a threat model similar to traditional cyber attacks. Attack forensics, a critical counter-measure for traditional cyber attacks, is hence of importance for defending model backdoor attacks. In this paper, we propose a novel model backdoor forensics technique. Given a few attack samples such as inputs with backdoor triggers, which may represent different types of backdoors, our technique automatically decomposes them to clean inputs and the corresponding triggers. It then clusters the triggers based on their properties to allow automatic attack categorization…
In this work, we propose the first backdoor attack to graph neural networks (GNN). Specifically, we propose a \emph{subgraph based backdoor attack} to GNN for graph classification. In our backdoor attack, a GNN classifier predicts an attacker-chosen target label for a testing graph once a predefined subgraph is injected to the testing graph. Our empirical results on three real-world graph datasets show that our backdoor attacks are effective with a small impact on a GNN's prediction accuracy for clean testing graphs. Moreover, we generalize a randomized smoothing based certified defense to def…
