In recent years, Deep Learning(DL) techniques have been extensively deployed for computer vision tasks, particularly visual classification problems, where new algorithms reported to achieve or even surpass the human performance. While many recent works demonstrated that DL models are vulnerable to adversarial examples. Fortunately, generating adversarial examples usually requires white-box access to the victim model, and real-world cloud-based image classification services are more complex than white-box classifier,the architecture and parameters of DL models on cloud platforms cannot be obtained by the attacker. The attacker can only access the APIs opened by cloud platforms. Thus, keeping models in the cloud can usually give a (false) sense of security. In this paper, we mainly focus on studying the security of real-world cloud-based image classification services. Specifically, (1) We propose a novel attack method, Fast Featuremap Loss PGD (FFL-PGD) attack based on Substitution model, which achieves a high bypass rate with a very limited number of queries. Instead of millions of queries in previous studies, our method finds the adversarial examples using only two queries per image; and (2) we make the first attempt to conduct an extensive empirical study of black-box attacks against real-world cloud-based classification services. Through evaluations on four popular cloud platforms including Amazon, Google, Microsoft, Clarifai, we demonstrate that FFL-PGD attack has a success rate over 90\% among different classification services. (3) We discuss the possible defenses to address these security challenges in cloud-based classification services. Our defense technology is mainly divided into model training stage and image preprocessing stage.
Fuzhou University（福州大学）；City University of Macau（澳门城市大学）；Peking University（北京大学）
Transferability of adversarial examples is of critical importance to launch black-box adversarial attacks, where attackers are only allowed to access the output of the target model. However, under such a challenging but practical setting, the crafted adversarial examples are always prone to overfitting to the proxy model employed, presenting poor transferability. In this paper, we suggest alleviating the overfitting issue from a novel perspective, i.e., designing a fitted model architecture. Specifically, delving the bottom of the cause of poor transferability, we arguably decompose and recons…
With the development of adversarial attacks, adversairal examples have been widely used to enhance the robustness of the training models on deep neural networks. Although considerable efforts of adversarial attacks on improving the transferability of adversarial examples have been developed, the attack success rate of the transfer-based attacks on the surrogate model is much higher than that on victim model under the low attack strength (e.g., the attack strength $ε=8/255$). In this paper, we first systematically investigated this issue and found that the enormous difference of attack success …
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；JD Explore Academy（京东探索研究院）；Tencent AI Lab（腾讯人工智能实验室）
Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples, which can produce erroneous predictions by injecting imperceptible perturbations. In this work, we study the transferability of adversarial examples, which is significant due to its threat to real-world applications where model architecture or parameters are usually unknown. Many existing works reveal that the adversarial examples are likely to overfit the surrogate model that they are generated from, limiting its transfer attack performance against different target models. To mitigate the overfitting of the …
Improving Transferability of Adversarial Examples via Bayesian Attacks
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学学院）；Department of Computer Science, University of California, Davis（加州大学戴维斯分校计算机科学系）
the University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Knowledge transferability, or transfer learning, has been widely adopted to allow a pre-trained model in the source domain to be effectively adapted to downstream tasks in the target domain. It is thus important to explore and understand the factors affecting knowledge transferability. In this paper, as the first work, we analyze and demonstrate the connections between knowledge transferability and another important phenomenon--adversarial transferability, \emph{i.e.}, adversarial examples generated against one model can be transferred to attack other models. Our theoretical studies show that …
ByteDance AI Lab（字节跳动人工智能实验室）；University of California, Davis（加州大学戴维斯分校）
The vulnerability of deep neural networks (DNNs) to adversarial examples has drawn great attention from the community. In this paper, we study the transferability of such examples, which lays the foundation of many black-box attacks on DNNs. We revisit a not so new but definitely noteworthy hypothesis of Goodfellow et al.'s and disclose that the transferability can be enhanced by improving the linearity of DNNs in an appropriate manner. We introduce linear backpropagation (LinBP), a method that performs backpropagation in a more linear fashion using off-the-shelf attacks that exploit gradients…
Although many methods have been proposed to enhance the transferability of adversarial perturbations, these methods are designed in a heuristic manner, and the essential mechanism for improving adversarial transferability is still unclear. This paper summarizes the common mechanism shared by twelve previous transferability-boosting methods in a unified view, i.e., these methods all reduce game-theoretic interactions between regional adversarial perturbations. To this end, we focus on the attacking utility of all interactions between regional adversarial perturbations, and we first discover and…
University of Luxembourg（卢森堡大学）；University of California, Berkeley（加州大学伯克利分校）
We propose transferability from Large Geometric Vicinity (LGV), a new technique to increase the transferability of black-box adversarial attacks. LGV starts from a pretrained surrogate model and collects multiple weight sets from a few additional training epochs with a constant and high learning rate. LGV exploits two geometric properties that we relate to transferability. First, models that belong to a wider weight optimum are better surrogates. Second, we identify a subspace able to generate an effective surrogate ensemble among this wider optimum. Through extensive experiments, we show that…
Adversarial examples are of wide concern due to their impact on the reliability of contemporary machine learning systems. Effective adversarial examples are mostly found via white-box attacks. However, in some cases they can be transferred across models, thus enabling them to attack black-box models. In this work we evaluate the transferability of three adversarial attacks - the Fast Gradient Sign Method, the Basic Iterative Method, and the Carlini & Wagner method, across two classes of models - the VGG class(using VGG16, VGG19 and an ensemble of VGG16 and VGG19), and the Inception class(Incep…
Huazhong University of Science and Technology（华中科技大学）；Cornell University（康奈尔大学）
The black-box adversarial attack has attracted impressive attention for its practical use in the field of deep learning security. Meanwhile, it is very challenging as there is no access to the network architecture or internal weights of the target model. Based on the hypothesis that if an example remains adversarial for multiple models, then it is more likely to transfer the attack capability to other models, the ensemble-based adversarial attack methods are efficient and widely used for black-box attacks. However, ways of ensemble attack are rather less investigated, and existing ensemble att…
Boosting Adversarial Transferability via Residual Perturbation Attack
School of Cyber Security and Computer（网络安全与计算机学院）；College of Information and Science Technology（信息与科学学院）；School of Computer and Information Engineering（计算机与信息工程学院）
Boosting Adversarial Transferability with Spatial Adversarial Alignment
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；College of Computer Science and Artificial Intelligence, Fudan University（计算机科学与人工智能学院，复旦大学）；China Southern Power Grid Artificial Intelligence Technology Co., Ltd.（中国南方电网人工智能技术有限公司）；School of Computer Science, Peking University（北京大学计算机学院）；vivo Mobile Communication Co., Ltd.（vivo移动通信有限公司）
Neural networks are vulnerable to adversarial examples, malicious inputs crafted to fool trained models. Adversarial examples often exhibit black-box transfer, meaning that adversarial examples for one model can fool another model. However, adversarial examples are typically overfit to exploit the particular architecture and feature representation of a source model, resulting in sub-optimal black-box transfer attacks to other target models. We introduce the Intermediate Level Attack (ILA), which attempts to fine-tune an existing adversarial example for greater black-box transferability by incr…
In recent years, convolutional neural networks (CNNs) have been widely used by researchers to perform forensic tasks such as image tampering detection. At the same time, adversarial attacks have been developed that are capable of fooling CNN-based classifiers. Understanding the transferability of adversarial attacks, i.e. an attacks ability to attack a different CNN than the one it was trained against, has important implications for designing CNNs that are resistant to attacks. While attacks on object recognition CNNs are believed to be transferrable, recent work by Barni et al. has shown that…
Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability
School of Cyber Science and Engineering, Huazhong University of Science and Technology（电子科学与工程学院，华中科技大学）；School of Software Engineering, Huazhong University of Science and Technology（软件工程学院，华中科技大学）；School of Information and Communication Technology, Griffith University（信息与通信技术学院，格里菲斯大学）；School of Computer Science and Technology, Huazhong University of Science and Technology（计算机科学与技术学院，华中科技大学）
Raw deep neural network (DNN) performance is not enough; in real-world settings, computational load, training efficiency and adversarial security are just as or even more important. We propose to simultaneously tackle Performance, Efficiency, and Robustness, using our proposed algorithm Q-TART, Quickly Train for Adversarial Robustness and in-Transferability. Q-TART follows the intuition that samples highly susceptible to noise strongly affect the decision boundaries learned by DNNs, which in turn degrades their performance and adversarial susceptibility. By identifying and removing such sample…
Syracuse University（雪城大学）；University of Michigan（密歇根大学）；UCLA（加州大学洛杉矶分校）；Duke University（杜克大学）
Neural networks are known to be vulnerable to carefully crafted adversarial examples, and these malicious samples often transfer, i.e., they remain adversarial even against other models. Although great efforts have been delved into the transferability across models, surprisingly, less attention has been paid to the cross-task transferability, which represents the real-world cybercriminal's situation, where an ensemble of different defense/detection mechanisms need to be evaded all at once. In this paper, we investigate the transferability of adversarial examples across a wide range of real-wor…
Rutgers University - School of Engineering（罗格斯大学工程学院）；MIT-IBM Watson AI Lab, IBM Research（IBM研究院麻省理工学院-IBM沃森人工智能实验室）
Deep neural networks (DNNs) are vulnerable to adversarial attack despite their tremendous success in many AI fields. Adversarial attack is a method that causes the intended misclassfication by adding imperceptible perturbations to legitimate inputs. Researchers have developed numerous types of adversarial attack methods. However, from the perspective of practical deployment, these methods suffer from several drawbacks such as long attack generating time, high memory cost, insufficient robustness and low transferability. We propose a Content-aware Adversarial Attack Generator (CAG) to achieve r…
Recent studies have shown that adversarial examples hand-crafted on one white-box model can be used to attack other black-box models. Such cross-model transferability makes it feasible to perform black-box attacks, which has raised security concerns for real-world DNNs applications. Nevertheless, existing works mostly focus on investigating the adversarial transferability across different deep models that share the same modality of input data. The cross-modal transferability of adversarial perturbation has never been explored. This paper investigates the transferability of adversarial perturba…
Tokyo Metropolitan University（东京都立大学）；National Institute of Informatics (NII)（国立信息学研究所）
Deep neural networks (DNNs) are well known to be vulnerable to adversarial examples (AEs). In addition, AEs have adversarial transferability, namely, AEs generated for a source model fool other (target) models. In this paper, we investigate the transferability of models encrypted for adversarially robust defense for the first time. To objectively verify the property of transferability, the robustness of models is evaluated by using a benchmark attack method, called AutoAttack. In an image-classification experiment, the use of encrypted models is confirmed not only to be robust against AEs but …
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）；University of Science, VNU-HCM（越南国立大学胡志明市分校科学大学）；Vietnam National University, Ho Chi Minh City（越南国立大学胡志明市分校）
Deep neural networks are vulnerable to adversarial examples (AEs), which have adversarial transferability: AEs generated for the source model can mislead another (target) model's predictions. However, the transferability has not been understood in terms of to which class target model's predictions were misled (i.e., class-aware transferability). In this paper, we differentiate the cases in which a target model predicts the same wrong class as the source model ("same mistake") or a different wrong class ("different mistake") to analyze and provide an explanation of the mechanism. We find that (…
Huazhong University of Science and Technology（华中科技大学）；Nanyang Technological University（南洋理工大学）；Microsoft Research Asia（微软亚洲研究院）
Deep neural networks are known to be extremely vulnerable to adversarial examples under white-box setting. Moreover, the malicious adversaries crafted on the surrogate (source) model often exhibit black-box transferability on other models with the same learning task but having different architectures. Recently, various methods are proposed to boost the adversarial transferability, among which the input transformation is one of the most effective approaches. We investigate in this direction and observe that existing transformations are all applied on a single image, which might limit the advers…
Vanderbilt University（范德堡大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
Despite the considerable success of convolutional neural networks in a broad array of domains, recent research has shown these to be vulnerable to small adversarial perturbations, commonly known as adversarial examples. Moreover, such examples have shown to be remarkably portable, or transferable, from one model to another, enabling highly successful black-box attacks. We explore this issue of transferability and robustness from two dimensions: first, considering the impact of conventional $l_p$ regularization as well as replacing the top layer with a linear support vector machine (SVM), and s…
RealAI（瑞莱智慧）；Ant Financial（蚂蚁金服）；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Nanyang Technological University（南洋理工大学）
Face recognition is greatly improved by deep convolutional neural networks (CNNs). Recently, these face recognition models have been used for identity authentication in security sensitive applications. However, deep CNNs are vulnerable to adversarial patches, which are physically realizable and stealthy, raising new security concerns on the real-world applications of these models. In this paper, we evaluate the robustness of face recognition models using adversarial patches based on transferability, where the attacker has limited accessibility to the target models. First, we extend the existin…
Neural networks are vulnerable to adversarial examples, malicious inputs crafted to fool trained models. Adversarial examples often exhibit black-box transfer, meaning that adversarial examples for one model can fool another model. However, adversarial examples may be overfit to exploit the particular architecture and feature representation of a source model, resulting in sub-optimal black-box transfer attacks to other target models. This leads us to introduce the Intermediate Level Attack (ILA), which attempts to fine-tune an existing adversarial example for greater black-box transferability …
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Microsoft Research Asia（微软亚洲研究院）
Deep learning models are known to be vulnerable to adversarial examples crafted by adding human-imperceptible perturbations on benign images. Many existing adversarial attack methods have achieved great white-box attack performance, but exhibit low transferability when attacking other models. Various momentum iterative gradient-based methods are shown to be effective to improve the adversarial transferability. In what follows, we propose an enhanced momentum iterative gradient-based method to further enhance the adversarial transferability. Specifically, instead of only accumulating the gradie…
TranSegPGD: Improving Transferability of Adversarial Examples on Semantic Segmentation
Nanyang Technological University（南洋理工大学）；University of Oxford（牛津大学）；Yanshan University（燕山大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
The security of models based on new architectures such as MLP-Mixer and ViTs needs to be studied urgently. However, most of the current researches are mainly aimed at the adversarial attack against ViTs, and there is still relatively little adversarial work on MLP-mixer. We propose an adversarial attack method against MLP-Mixer called Maxwell's demon Attack (MA). MA breaks the channel-mixing and token-mixing mechanism of MLP-Mixer by controlling the part input of MLP-Mixer's each Mixer layer, and disturbs MLP-Mixer to obtain the main information of images. Our method can mask the part input of…
University of Padova（帕多瓦大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. The latter property may seem counter-intuitive as it is widely accepted by the community that classification models should only capture the minimal information (features) required for the task. Motivated by this discrepancy, we investigate the dual relationship between Adversarial Training and Information Theory. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-…
Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；PCL Research Center of Networks and Communications, Peng Cheng Laboratory（鹏城实验室网络通信PCL研究中心）；The University of Melbourne（墨尔本大学）
Skip connections are an essential component of current state-of-the-art deep neural networks (DNNs) such as ResNet, WideResNet, DenseNet, and ResNeXt. Despite their huge success in building deeper and more powerful DNNs, we identify a surprising security weakness of skip connections in this paper. Use of skip connections allows easier generation of highly transferable adversarial examples. Specifically, in ResNet-like (with skip connections) neural networks, gradients can backpropagate through either skip connections or residual modules. We find that using more gradients from the skip connecti…
Improving Adversarial Transferability with Neighbourhood Gradient Information
Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理关键实验室）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Engineering Research Center of AI & Robotics（上海人工智能与机器人工程研究中心）；Academy for Engineering & Technology, Fudan University（复旦大学工程与技术学院）；Engineering Research Center of Robotics, Ministry of Education（教育部机器人工程研究中心）
University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）
Despite the recent advancements in deploying neural networks for image classification, it has been found that adversarial examples are able to fool these models leading them to misclassify the images. Since these models are now being widely deployed, we provide an insight on the threat of these adversarial examples by evaluating their characteristics and transferability to more complex models that utilize Image Classification as a subtask. We demonstrate the ineffectiveness of adversarial examples when applied to Instance Segmentation & Object Detection models. We show that this ineffectivenes…
SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Sun Yat-sen University（中山大学）
Sabanci University（萨班哲大学）；University of Padua（帕多瓦大学）；Moulay Ismail University of Meknes（梅克内斯穆莱·伊斯梅尔大学）；Florida International University（佛罗里达国际大学）
Convolutional Neural Networks (CNNs) models are one of the most frequently used deep learning networks, and extensively used in both academia and industry. Recent studies demonstrated that adversarial attacks against such models can maintain their effectiveness even when used on models other than the one targeted by the attacker. This major property is known as transferability, and makes CNNs ill-suited for security applications. In this paper, we provide the first comprehensive study which assesses the robustness of CNN-based models for computer networks against adversarial transferability. F…
Ghent University（根特大学）；VIB Inflammation Research Center（VIB炎症研究中心）；imec（比利时微电子研究中心）
Regional adversarial attacks often rely on complicated methods for generating adversarial perturbations, making it hard to compare their efficacy against well-known attacks. In this study, we show that effective regional perturbations can be generated without resorting to complex methods. We develop a very simple regional adversarial perturbation attack method using cross-entropy sign, one of the most commonly used losses in adversarial machine learning. Our experiments on ImageNet with multiple models reveal that, on average, $76\%$ of the generated adversarial examples maintain model-to-mode…
Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；University of Oxford（牛津大学）；Drexel University（德雷塞尔大学）；Beijing University of Technology（北京理工大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Xi’an Jiaotong University（西安交通大学）
