Towards Backdoor Attacks and Defense in Robust Machine Learning Models
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学跨学科安全、可靠性与信任中心（SnT））；Singapore University of Technology and Design（新加坡科技设计大学）
The introduction of robust optimisation has pushed the state-of-the-art in defending against adversarial attacks. Notably, the state-of-the-art projected gradient descent (PGD)-based training method has been shown to be universally and reliably effective in defending against adversarial inputs. This robustness approach uses PGD as a reliable and universal "first-order adversary". However, the behaviour of such optimisation has not been studied in the light of a fundamentally different class of attacks called backdoors. In this paper, we study how to inject and defend against backdoor attacks for robust models trained using PGD-based robust optimisation. We demonstrate that these models are susceptible to backdoor attacks. Subsequently, we observe that backdoors are reflected in the feature representation of such models. Then, this observation is leveraged to detect such backdoor-infected models via a detection technique called AEGIS. Specifically, given a robust Deep Neural Network (DNN) that is trained using PGD-based first-order adversarial training approach, AEGIS uses feature clustering to effectively detect whether such DNNs are backdoor-infected or clean.
In our evaluation of several visible and hidden backdoor triggers on major classification tasks using CIFAR-10, MNIST and FMNIST datasets, AEGIS effectively detects PGD-trained robust DNNs infected with backdoors. AEGIS detects such backdoor-infected models with 91.6% accuracy (11 out of 12 tested models), without any false positives. Furthermore, AEGIS detects the targeted class in the backdoor-infected model with a reasonably low (11.1%) false positive rate. Our investigation reveals that salient features of adversarially robust DNNs could be promising to break the stealthy nature of backdoor attacks.
Deep neural networks for image classification are well-known to be vulnerable to adversarial attacks. One such attack that has garnered recent attention is the adversarial backdoor attack, which has demonstrated the capability to perform targeted misclassification of specific examples. In particular, backdoor attacks attempt to force a model to learn spurious relations between backdoor trigger patterns and false labels. In response to this threat, numerous defensive measures have been proposed; however, defenses against backdoor attacks focus on backdoor pattern detection, which may be unrelia…
Modern machine learning increasingly requires training on a large collection of data from multiple sources, not all of which can be trusted. A particularly concerning scenario is when a small fraction of poisoned data changes the behavior of the trained model when triggered by an attacker-specified watermark. Such a compromised model will be deployed unnoticed as the model is accurate otherwise. There have been promising attempts to use the intermediate representations of such a model to separate corrupted examples from clean ones. However, these defenses work only when a certain spectral sign…
Xi’an Jiaotong University（西安交通大学）；Alibaba Group（阿里巴巴集团）；Wormpex AI Research（Wormpex AI研究院）
Deep neural networks (DNNs) are known to be vulnerable to both backdoor attacks as well as adversarial attacks. In the literature, these two types of attacks are commonly treated as distinct problems and solved separately, since they belong to training-time and inference-time attacks respectively. However, in this paper we find an intriguing connection between them: for a model planted with backdoors, we observe that its adversarial examples have similar behaviors as its triggered images, i.e., both activate the same subset of DNN neurons. It indicates that planting a backdoor into a model wil…
Fuzhou University（福州大学）；East China Normal University（华东师范大学）
The opacity of neural networks leads their vulnerability to backdoor attacks, where hidden attention of infected neurons is triggered to override normal predictions to the attacker-chosen ones. In this paper, we propose a novel backdoor defense method to mark and purify the infected neurons in the backdoored neural networks. Specifically, we first define a new metric, called benign salience. By combining the first-order gradient to retain the connections between neurons, benign salience can identify the infected neurons with higher accuracy than the commonly used metric in backdoor defense. Th…
Princeton University（普林斯顿大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Recent studies revealed that deep learning is susceptible to backdoor poisoning attacks. An adversary can embed a hidden backdoor into a model to manipulate its predictions by only modifying a few training data, without controlling the training process. Currently, a tangible signature has been widely observed across a diverse set of backdoor poisoning attacks -- models trained on a poisoned dataset tend to learn separable latent representations for poison and clean samples. This latent separation is so pervasive that a family of backdoor defenses directly take it as a default assumption (dubbe…
School of Cyber Science and Technology Sun Yat-Sen University（中山大学网络空间安全学院）；Department of Computer Science National University of Singapore（新加坡国立大学计算机科学系）
Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images
University of West Florida（西佛罗里达大学）；Florida Institute For Human and Machine Cognition (IHMC)（佛罗里达人机认知研究所）；Center for Cybersecurity, University of West Florida（西佛罗里达大学网络安全中心）
Backdoor defenses have been studied to alleviate the threat of deep neural networks (DNNs) being backdoor attacked and thus maliciously altered. Since DNNs usually adopt some external training data from an untrusted third party, a robust backdoor defense strategy during the training stage is of importance. We argue that the core of training-time defense is to select poisoned samples and to handle them properly. In this work, we summarize the training-time defenses from a unified framework as splitting the poisoned dataset into two data pools. Under our framework, we propose an adaptively split…
REFINE: Inversion-Free Backdoor Defense via Model Reprogramming
Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）；Nanyang Technological University（南洋理工大学）；IBM Research（IBM研究院）
A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
Singapore Management University（新加坡管理大学）；China University of Mining and Technology（中国矿业大学）；The University of Melbourne（墨尔本大学）；Anhui University（安徽大学）；Xi’an Jiaotong University（西安交通大学）；Sun Yat-sen University（中山大学）
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
Recent studies have demonstrated that deep neural networks (DNNs) are vulnerable to backdoor attacks during the training process. Specifically, the adversaries intend to embed hidden backdoors in DNNs so that malicious model predictions can be activated through pre-defined trigger patterns. In this paper, we explore the backdoor mechanism from the angle of the model structure. We select the skip connection for discussions, inspired by the understanding that it helps the learning of model `shortcuts' where backdoor triggers are usually easier to be learned. Specifically, we demonstrate that the…
Pennsylvania State University（宾夕法尼亚州立大学）；Shanghai Jiao Tong University（上海交通大学）
Backdoor attacks (BA) are an emerging threat to deep neural network classifiers. A classifier being attacked will predict to the attacker's target class when a test sample from a source class is embedded with the backdoor pattern (BP). Recently, the first BA against point cloud (PC) classifiers was proposed, creating new threats to many important applications including autonomous driving. Such PC BAs are not detectable by existing BA defenses due to their special BP embedding mechanism. In this paper, we propose a reverse-engineering defense that infers whether a PC classifier is backdoor atta…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Saudi Data and Artificial Intelligence Authority (SDAIA)（沙特数据与人工智能管理局）
Deep neural networks (DNNs) are vulnerable to a class of attacks called "backdoor attacks", which create an association between a backdoor trigger and a target label the attacker is interested in exploiting. A backdoored DNN performs well on clean test images, yet persistently predicts an attacker-defined label for any sample in the presence of the backdoor trigger. Although backdoor attacks have been extensively studied in the image domain, there are very few works that explore such attacks in the video domain, and they tend to conclude that image backdoor attacks are less effective in the vi…
Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Recent studies have revealed that deep neural networks (DNNs) are vulnerable to backdoor attacks, where attackers embed hidden backdoors in the DNN model by poisoning a few training samples. The attacked model behaves normally on benign samples, whereas its prediction will be maliciously changed when the backdoor is activated. We reveal that poisoned samples tend to cluster together in the feature space of the attacked DNN model, which is mostly due to the end-to-end supervised training paradigm. Inspired by this observation, we propose a novel backdoor defense via decoupling the original end-…
Learning Normal Diffusion Dynamics for Backdoor Defense in Text-to-Image Models
Geely（吉利）；Hunan University（湖南大学）；East China Normal University（华东师范大学）；University of Adelaide（阿德莱德大学）；The Hong Kong University of Science and Technology（香港科技大学）；China University of Petroleum (East China)（中国石油大学（华东））
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；University of Birmingham（伯明翰大学）；University of Lausanne（洛桑大学）
Backdoor attacks represent one of the major threats to machine learning models. Various efforts have been made to mitigate backdoors. However, existing defenses have become increasingly complex and often require high computational resources or may also jeopardize models' utility. In this work, we show that fine-tuning, one of the most common and easy-to-adopt machine learning training operations, can effectively remove backdoors from machine learning models while maintaining high model utility. Extensive experiments over three machine learning paradigms show that fine-tuning and our newly prop…
Backdoor attacks embed hidden malicious behaviors into deep learning models, which only activate and cause misclassifications on model inputs containing a specific trigger. Existing works on backdoor attacks and defenses, however, mostly focus on digital attacks that use digitally generated patterns as triggers. A critical question remains unanswered: can backdoor attacks succeed using physical objects as triggers, thus making them a credible threat against deep learning systems in the real world? We conduct a detailed empirical study to explore this question for facial recognition, a critical…
Self-supervised learning in computer vision aims to pre-train an image encoder using a large amount of unlabeled images or (image, text) pairs. The pre-trained image encoder can then be used as a feature extractor to build downstream classifiers for many downstream tasks with a small amount of or no labeled training data. In this work, we propose BadEncoder, the first backdoor attack to self-supervised learning. In particular, our BadEncoder injects backdoors into a pre-trained image encoder such that the downstream classifiers built based on the backdoored image encoder for different downstre…
Cert-SSBD: Certified Backdoor Defense with Sample-Specific Smoothing Noises
School of Control and Computer Engineering, North China Electric Power University（控制与计算机工程学院，华北电力大学）；Research Institute, China Unicom（中国unicom研究院）；National Engineering Research Center of Next Generation Internet Broadband Service Application（下一代互联网宽带服务应用国家工程研究中心）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
CISPA Helmholtz Center for Information Security（CISPA 亥姆霍兹信息安全研究中心）
Backdoor attack against deep neural networks is currently being profoundly investigated due to its severe security consequences. Current state-of-the-art backdoor attacks require the adversary to modify the input, usually by adding a trigger to it, for the target model to activate the backdoor. This added trigger not only increases the difficulty of launching the backdoor attack in the physical world, but also can be easily detected by multiple defense mechanisms. In this paper, we present the first triggerless backdoor attack against deep neural networks, where the adversary does not need to …
BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Beihang University（北京航空航天大学）；Sun Yat-sen University-Shenzhen（中山大学（深圳））
Nanjing University（南京大学）；The University of Queensland（昆士兰大学）；Purdue University（普渡大学）；Pennsylvania State University（宾夕法尼亚州立大学）
Backdoor attacks have been shown to be a serious security threat against deep learning models, and detecting whether a given model has been backdoored becomes a crucial task. Existing defenses are mainly built upon the observation that the backdoor trigger is usually of small size or affects the activation of only a few neurons. However, the above observations are violated in many cases especially for advanced backdoor attacks, hindering the performance and applicability of the existing defenses. In this paper, we propose a backdoor defense DTInspector built upon a new observation. That is, an…
University of Maryland（马里兰大学）；Massachussetts Institute of Technology（麻省理工学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）
As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance. The absence of trustworthy human supervision over the data collection process exposes organizations to security vulnerabilities; training data can be manipulated to control and degrade the downstream behaviors of learned models. The goal of this work is to systematically categorize and discuss a wide range of dataset vulnerabilities and exploits, approaches for defending against these t…
Recent studies have shown that DNNs can be compromised by backdoor attacks crafted at training time. A backdoor attack installs a backdoor into the victim model by injecting a backdoor pattern into a small proportion of the training data. At test time, the victim model behaves normally on clean test data, yet consistently predicts a specific (likely incorrect) target class whenever the backdoor pattern is present in a test example. While existing backdoor attacks are effective, they are not stealthy. The modifications made on training data or labels are often suspicious and can be easily detec…
Fudan University（复旦大学）；The University of Melbourne（墨尔本大学）
Deep neural networks (DNNs) are vulnerable to backdoor attacks which can hide backdoor triggers in DNNs by poisoning training data. A backdoored model behaves normally on clean test images, yet consistently predicts a particular target class for any test examples that contain the trigger pattern. As such, backdoor attacks are hard to detect, and have raised severe security concerns in real-world applications. Thus far, backdoor research has mostly been conducted in the image domain with image classification models. In this paper, we show that existing image backdoor attacks are far less effect…
Imperceptible and Multi-channel Backdoor Attack against Deep Neural Networks
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics（南京航空航天大学电子信息工程学院）
In recent years, machine learning models have been shown to be vulnerable to backdoor attacks. Under such attacks, an adversary embeds a stealthy backdoor into the trained model such that the compromised models will behave normally on clean inputs but will misclassify according to the adversary's control on maliciously constructed input with a trigger. While these existing attacks are very effective, the adversary's capability is limited: given an input, these attacks can only cause the model to misclassify toward a single pre-defined or target class. In contrast, this paper exploits a novel b…
Nanjing University（南京大学）；Purdue University（普渡大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Pennsylvania State University（宾夕法尼亚州立大学）
Backdoor attacks have been shown to be a serious threat against deep learning systems such as biometric authentication and autonomous driving. An effective backdoor attack could enforce the model misbehave under certain predefined conditions, i.e., triggers, but behave normally otherwise. However, the triggers of existing attacks are directly injected in the pixel space, which tend to be detectable by existing defenses and visually identifiable at both training and inference stages. In this paper, we propose a new backdoor attack FTROJAN through trojaning the frequency domain. The key intuitio…
