As the curation of data for machine learning becomes increasingly automated, dataset tampering is a mounting threat. Backdoor attackers tamper with training data to embed a vulnerability in models that are trained on that data. This vulnerability is then activated at inference time by placing a "trigger" into the model's input. Typical backdoor attacks insert the trigger directly into the training data, although the presence of such an attack may be visible upon inspection. In contrast, the Hidden Trigger Backdoor Attack achieves poisoning without placing a trigger into the training data at all. However, this hidden trigger attack is ineffective at poisoning neural networks trained from scratch. We develop a new hidden trigger attack, Sleeper Agent, which employs gradient matching, data selection, and target model re-training during the crafting process. Sleeper Agent is the first hidden trigger backdoor attack to be effective against neural networks trained from scratch. We demonstrate its effectiveness on ImageNet and in black-box settings. Our implementation code can be found at https://github.com/hsouri/Sleeper-Agent.
Generating Potent Poisons and Backdoors from Scratch with Guided Diffusion
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland（马里兰大学）；Google（谷歌）；ELLIS Institute Tübingen（埃利斯研究所蒂宾根分支）；Tübingen AI Center（蒂宾根人工智能中心）；New York University（纽约大学）
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Anthropic；Redwood Research（红木研究）；Mila Quebec AI Institute（Mila魁北克人工智能研究所）；University of Oxford（牛津大学）；Alignment Research Center（对齐研究中心）；Open Philanthropy（开放慈善）；Apart Research
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Michigan State University（密歇根州立大学）；Amazon Web Services（亚马逊云科技）
Deep neural networks (DNNs) are vulnerable to backdoor attacks. Previous works have shown it extremely challenging to unlearn the undesired backdoor behavior from the network, since the entire network can be affected by the backdoor samples. In this paper, we propose a brand-new backdoor defense strategy, which makes it much easier to remove the harmful influence of backdoor samples from the model. Our defense strategy, \emph{Trap and Replace}, consists of two stages. In the first stage, we bait and trap the backdoors in a small and easy-to-replace subnetwork. Specifically, we add an auxiliary…
University of Cambridge（剑桥大学）；Vector Institute（向量研究所）；University of Toronto（多伦多大学）
Machine learning is vulnerable to adversarial manipulation. Previous literature has demonstrated that at the training stage attackers can manipulate data and data sampling procedures to control model behaviour. A common attack goal is to plant backdoors i.e. force the victim model to learn to recognise a trigger known only by the adversary. In this paper, we introduce a new class of backdoor attacks that hide inside model architectures i.e. in the inductive bias of the functions used to train. These backdoors are simple to implement, for instance by publishing open-source code for a backdoored…
Since training a large-scale backdoored model from scratch requires a large training dataset, several recent attacks have considered to inject backdoors into a trained clean model without altering model behaviors on the clean data. Previous work finds that backdoors can be injected into a trained clean model with Adversarial Weight Perturbation (AWP). Here AWPs refers to the variations of parameters that are small in backdoor learning. In this work, we observe an interesting phenomenon that the variations of parameters are always AWPs when tuning the trained clean model to inject backdoors. We…
The Pennsylvania State University（宾夕法尼亚州立大学）；Zhejiang University（浙江大学）
This paper asks the intriguing question: is it possible to exploit neural architecture search (NAS) as a new attack vector to launch previously improbable attacks? Specifically, we present EVAS, a new attack that leverages NAS to find neural architectures with inherent backdoors and exploits such vulnerability using input-aware triggers. Compared with existing attacks, EVAS demonstrates many interesting properties: (i) it does not require polluting training data or perturbing model parameters; (ii) it is agnostic to downstream fine-tuning or even re-training from scratch; (iii) it naturally ev…
German Research Center for Artificial Intelligence（德国人工智能研究中心）；Technical University of Darmstadt（达姆施塔特工业大学）；TU Dortmund University（多特蒙德工业大学）；Center for Trustworthy Data Science and Security, University Alliance Ruhr（鲁尔大学联盟可信数据科学与安全中心）；Hessian Center for AI (hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science TU Darmstadt（达姆施塔特工业大学认知科学中心）
Federated Learning (FL) is a new machine learning framework, which enables millions of participants to collaboratively train machine learning model without compromising data privacy and security. Due to the independence and confidentiality of each client, FL does not guarantee that all clients are honest by design, which makes it vulnerable to adversarial attack naturally. In this paper, we focus on dynamic backdoor attacks under FL setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining a good performance on the main task, current…
Pennsylvania State University（宾夕法尼亚州立大学）；Zhejiang University（浙江大学）；Hong Kong Polytechnic University（香港理工大学）
Neural backdoors represent one primary threat to the security of deep learning systems. The intensive research has produced a plethora of backdoor attacks/defenses, resulting in a constant arms race. However, due to the lack of evaluation benchmarks, many critical questions remain under-explored: (i) what are the strengths and limitations of different attacks/defenses? (ii) what are the best practices to operate them? and (iii) how can the existing attacks/defenses be further improved? To bridge this gap, we design and implement TROJANZOO, the first open-source platform for evaluating neural b…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Deep neural networks (DNN) have achieved remarkable performance in various fields. However, training a DNN model from scratch requires a lot of computing resources and training data. It is difficult for most individual users to obtain such computing resources and training data. Model copyright infringement is an emerging problem in recent years. For instance, pre-trained models may be stolen or abuse by illegal users without the authorization of the model owner. Recently, many works on protecting the intellectual property of DNN models have been proposed. In these works, embedding watermarks i…
SKLOIS, Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所智能信息处理实验室）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Machine unlearning has great significance in guaranteeing model security and protecting user privacy. Additionally, many legal provisions clearly stipulate that users have the right to demand model providers to delete their own data from training set, that is, the right to be forgotten. The naive way of unlearning data is to retrain the model without it from scratch, which becomes extremely time and resource consuming at the modern scale of deep neural networks. Other unlearning approaches by refactoring model or training data struggle to gain a balance between overhead and model usability. In…
IIIT Hyderabad（印度信息技术研究所海得拉巴分校）；ELLIS Institute Tübingen（埃利斯研究所蒂宾根分所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Oxford（牛津大学）；Tübingen AI Center（蒂宾根人工智能中心）；University of Copenhagen（哥本哈根大学）
