Adversarial attacks can compromise the robustness of real-world detection models. However, evaluating these models under real-world conditions poses challenges due to resource-intensive experiments. Virtual simulations offer an alternative, but the absence of standardized benchmarks hampers progress. Addressing this, we propose an innovative instant-level data generation pipeline using the CARLA simulator. Through this pipeline, we establish the Discrete and Continuous Instant-level (DCI) dataset, enabling comprehensive experiments involving three detection models and three physical adversarial attacks. Our findings highlight diverse model performances under adversarial conditions. Yolo v6 demonstrates remarkable resilience, experiencing just a marginal 6.59% average drop in average precision (AP). In contrast, the ASA attack yields a substantial 14.51% average AP reduction, twice the effect of other algorithms. We also note that static scenes yield higher recognition AP values, and outcomes remain relatively consistent across varying weather conditions. Intriguingly, our study suggests that advancements in adversarial attack algorithms may be approaching its ``limitation''.In summary, our work underscores the significance of adversarial attacks in real-world contexts and introduces the DCI dataset as a versatile benchmark. Our findings provide valuable insights for enhancing the robustness of detection models and offer guidance for future research endeavors in the realm of adversarial attacks.
This work conducts the first analysis on the robustness against adversarial attacks on self-supervised Vision Transformers trained using DINO. First, we evaluate whether features learned through self-supervision are more robust to adversarial attacks than those emerging from supervised learning. Then, we present properties arising for attacks in the latent space. Finally, we evaluate whether three well-known defense strategies can increase adversarial robustness in downstream tasks by only fine-tuning the classification head to provide robustness even in view of limited compute resources. Thes…
Rafael - Advanced Defense Systems Ltd.（拉斐尔先进防务系统有限公司）
In this paper, we present a proof of concept for adversarially attacking the image-based localization module of an autonomous vehicle. This attack aims to cause the vehicle to perform a wrong navigational decisions and prevent it from reaching a desired predefined destination in a simulated urban environment. A database of rendered images allowed us to train a deep neural network that performs a localization task and implement, develop and assess the adversarial pattern. Our tests show that using this adversarial attack we can prevent the vehicle from turning at a given intersection. This is d…
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Physical adversarial attacks in object detection have attracted increasing attention. However, most previous works focus on hiding the objects from the detector by generating an individual adversarial patch, which only covers the planar part of the vehicle's surface and fails to attack the detector in physical scenarios for multi-view, long-distance and partially occluded objects. To bridge the gap between digital attacks and physical attacks, we exploit the full 3D vehicle surface to propose a robust Full-coverage Camouflage Attack (FCA) to fool detectors. Specifically, we first try rendering…
A counter-intuitive property of convolutional neural networks (CNNs) is their inherent susceptibility to adversarial examples, which severely hinders the application of CNNs in security-critical fields. Adversarial examples are similar to original examples but contain malicious perturbations. Adversarial training is a simple and effective defense method to improve the robustness of CNNs to adversarial examples. The mechanisms behind adversarial examples and adversarial training are worth exploring. Therefore, this work investigates similarities and differences between normally trained CNNs (NT…
Sun Yat-Sen University（中山大学）；Imperial College London（帝国理工学院）
Although current deep learning techniques have yielded superior performance on various computer vision tasks, yet they are still vulnerable to adversarial examples. Adversarial training and its variants have been shown to be the most effective approaches to defend against adversarial examples. These methods usually regularize the difference between output probabilities for an adversarial and its corresponding natural example. However, it may have a negative impact if the model misclassifies a natural example. To circumvent this issue, we propose a novel adversarial training scheme that encoura…
In a modern vehicle, there are over seventy Electronics Control Units (ECUs). For an in-vehicle network, ECUs communicate with each other by following a standard communication protocol, such as Controller Area Network (CAN). However, an attacker can easily access the in-vehicle network to compromise ECUs through a WLAN or Bluetooth. Though there are various deep learning (DL) methods suggested for securing in-vehicle networks, recent studies on adversarial examples have shown that attackers can easily fool DL models. In this research, we further explore adversarial examples in an in-vehicle ne…
Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
In this paper, we tackle the issue of physical adversarial examples for object detectors in the wild. Specifically, we proposed to generate adversarial patterns to be applied on vehicle surface so that it's not recognizable by detectors in the photo-realistic Carla simulator. Our approach contains two main techniques, an Enlarge-and-Repeat process and a Discrete Searching method, to craft mosaic-like adversarial vehicle textures without access to neither the model weight of the detector nor a differential rendering procedure. The experimental results demonstrate the effectiveness of our approa…
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）；Southeast Missouri State University（东南密苏里州立大学）
Despite the high quality performance of the deep neural network in real-world applications, they are susceptible to minor perturbations of adversarial attacks. This is mostly undetectable to human vision. The impact of such attacks has become extremely detrimental in autonomous vehicles with real-time "safety" concerns. The black-box adversarial attacks cause drastic misclassification in critical scene elements such as road signs and traffic lights leading the autonomous vehicle to crash into other vehicles or pedestrians. In this paper, we propose a novel query-based attack method called Modi…
University of Michigan（密歇根大学）；University of California, Irvine（加州大学欧文分校）
Trajectory prediction is a critical component for autonomous vehicles (AVs) to perform safe planning and navigation. However, few studies have analyzed the adversarial robustness of trajectory prediction or investigated whether the worst-case prediction can still lead to safe planning. To bridge this gap, we study the adversarial robustness of trajectory prediction models by proposing a new adversarial attack that perturbs normal vehicle trajectories to maximize the prediction error. Our experiments on three models and three datasets show that the adversarial prediction increases the predictio…
City University of Hong Kong（香港城市大学）；University of Victoria（维多利亚大学）；University of Puerto Rico at Mayagüez（波多黎各大学马亚圭斯分校）
In recent years, many deep learning models have been adopted in autonomous driving. At the same time, these models introduce new vulnerabilities that may compromise the safety of autonomous vehicles. Specifically, recent studies have demonstrated that adversarial attacks can cause a significant decline in detection precision of deep learning-based 3D object detection models. Although driving safety is the ultimate concern for autonomous driving, there is no comprehensive study on the linkage between the performance of deep learning models and the driving safety of autonomous vehicles under adv…
AR-GAN: Generative Adversarial Network-Based Defense Method Against Adversarial Attacks on the Traffic Sign Classification System of Autonomous Vehicles
Progress in making neural networks more robust against adversarial attacks is mostly marginal, despite the great efforts of the research community. Moreover, the robustness evaluation is often imprecise, making it difficult to identify promising approaches. We analyze the classification decisions of 19 different state-of-the-art neural networks trained to be robust against adversarial attacks. Our findings suggest that current untargeted adversarial attacks induce misclassification towards only a limited amount of different classes. Additionally, we observe that both over- and under-confidence…
Deep neural networks are learning models having achieved state of the art performance in many fields like prediction, computer vision, language processing and so on. However, it has been shown that certain inputs exist which would not trick a human normally, but may mislead the model completely. These inputs are known as adversarial inputs. These inputs pose a high security threat when such models are used in real world applications. In this work, we have analyzed the resistance of three different classes of fully connected dense networks against the rarely tested non-gradient based adversaria…
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Key Lab. of Machine Perception, School of Artificial Intelligence, Peking University（北京大学人工智能学院机器感知重点实验室）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；University of California, Los Angeles（加州大学洛杉矶分校）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make …
Waabi；University of Toronto（多伦多大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Cornell University（康奈尔大学）
Modern self-driving perception systems have been shown to improve upon processing complementary inputs such as LiDAR with images. In isolation, 2D images have been found to be extremely vulnerable to adversarial attacks. Yet, there have been limited studies on the adversarial robustness of multi-modal models that fuse LiDAR features with image features. Furthermore, existing works do not consider physically realizable perturbations that are consistent across the input modalities. In this paper, we showcase practical susceptibilities of multi-sensor detection by placing an adversarial object on…
Adversarial examples (AEs) are images that can mislead deep neural network (DNN) classifiers via introducing slight perturbations into original images. This security vulnerability has led to vast research in recent years because it can introduce real-world threats into systems that rely on neural networks. Yet, a deep understanding of the characteristics of adversarial examples has remained elusive. We propose a new way of achieving such understanding through a recent development, namely, invertible neural models with Lipschitz continuous mapping functions from the input to the output. With th…
All Vehicles Can Lie: Efficient Adversarial Defense in Fully Untrusted-Vehicle Collaborative Perception via Pseudo-Random Bayesian Inference
School of Cyber Science and Engineering, Wuhan University（武汉大学网络科学与工程学院）；Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学先进技术网络研究院）
Deep neural networks are vulnerable to adversarial attacks. White-box adversarial attacks can fool neural networks with small adversarial perturbations, especially for large size images. However, keeping successful adversarial perturbations imperceptible is especially challenging for transfer-based black-box adversarial attacks. Often such adversarial examples can be easily spotted due to their unpleasantly poor visual qualities, which compromises the threat of adversarial attacks in practice. In this study, to improve the image quality of black-box adversarial examples perceptually, we propos…
Adversarial training has proven to be effective in hardening networks against adversarial examples. However, the gained robustness is limited by network capacity and number of training samples. Consequently, to build more robust models, it is common practice to train on widened networks with more parameters. To boost robustness, we propose a conditional normalization module to adapt networks when conditioned on input samples. Our adaptive networks, once adversarially trained, can outperform their non-adaptive counterparts on both clean validation accuracy and robustness. Our method is objectiv…
VCAT: Vulnerability-aware and Curiosity-driven Adversarial Training for Enhancing Autonomous Vehicle Robustness
State Key Laboratory of Intelligent Transportation Systems（智能交通系统国家重点实验室）；School of Transportation Science and Technology, Beihang University（北京航空航天大学交通科学与工程学院）；Department of Civil and Environmental Engineering, Rensselaer Polytechnic Institute（伦斯勒理工学院土木与环境工程系）
Deep learning models have a propensity for fitting the entire training set even with random labels, which requires memorization of every training sample. In this paper, we explore the memorization effect in adversarial training (AT) for promoting a deeper understanding of model capacity, convergence, generalization, and especially robust overfitting of the adversarially trained models. We first demonstrate that deep networks have sufficient capacity to memorize adversarial examples of training data with completely random labels, but not all AT algorithms can converge under the extreme circumst…
The transferability and robustness of adversarial examples are two practical yet important properties for black-box adversarial attacks. In this paper, we explore effective mechanisms to boost both of them from the perspective of network hierarchy, where a typical network can be hierarchically divided into output stage, intermediate stage and input stage. Since over-specialization of source model, we can hardly improve the transferability and robustness of the adversarial perturbations in the output stage. Therefore, we focus on the intermediate and input stages in this paper and propose a tra…
IBM Research（IBM研究院）；IBM Research Ireland（IBM爱尔兰研究院）；IBM Research India（IBM印度研究院）
Adversarial training shows promise as an approach for training models that are robust towards adversarial perturbation. In this paper, we explore some of the practical challenges of adversarial training. We present a sensitivity analysis that illustrates that the effectiveness of adversarial training hinges on the settings of a few salient hyperparameters. We show that the robustness surface that emerges across these salient parameters can be surprisingly complex and that therefore no effective one-size-fits-all parameter settings exist. We then demonstrate that we can use the same salient hyp…
University of Chinese Academy of Sciences（中国科学院大学）；Center for Research on Intelligent Perception and Computing, CASIA（中国科学院自动化研究所智能感知与计算研究中心）
Images synthesized by powerful generative adversarial network (GAN) based methods have drawn moral and privacy concerns. Although image forensic models have reached great performance in detecting fake images from real ones, these models can be easily fooled with a simple adversarial attack. But, the noise adding adversarial samples are also arousing suspicion. In this paper, instead of adding adversarial noise, we optimally search adversarial points on face manifold to generate anti-forensic fake face images. We iteratively do a gradient-descent with each small step in the latent space of a ge…
Peking University（北京大学）；Beijing Institute of Big Data Research (BIBDR)（北京大数据研究院）；Shanghai Jiaotong University（上海交通大学）
Most previous works usually explained adversarial examples from several specific perspectives, lacking relatively integral comprehension about this problem. In this paper, we present a systematic study on adversarial examples from three aspects: the amount of training data, task-dependent and model-specific factors. Particularly, we show that adversarial generalization (i.e. test accuracy on adversarial examples) for standard training requires more data than standard generalization (i.e. test accuracy on clean examples); and uncover the global relationship between generalization and robustness…
Centre for Intelligent Sensing, Queen Mary University of London（伦敦玛丽女王大学智能感知中心）；Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Autonomous Vehicles rely on accurate and robust sensor observations for safety critical decision-making in a variety of conditions. Fundamental building blocks of such systems are sensors and classifiers that process ultrasound, RADAR, GPS, LiDAR and camera signals~\cite{Khan2018}. It is of primary importance that the resulting decisions are robust to perturbations, which can take the form of different types of nuisances and data transformations, and can even be adversarial perturbations (APs). Adversarial perturbations are purposefully crafted alterations of the environment or of the sensory …
SenseTime（商汤科技）；Zhongguancun Laboratory（中关村实验室）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；JD Explore Academy（京东探索研究院）；Beihang University（北京航空航天大学）；Chinese Academy of Sciences（中国科学院）；Zhejiang University（浙江大学）
Adversarial training has been demonstrated to be one of the most effective remedies for defending adversarial examples, yet it often suffers from the huge robustness generalization gap on unseen testing adversaries, deemed as the adversarially robust generalization problem. Despite the preliminary understandings devoted to adversarially robust generalization, little is known from the architectural perspective. To bridge the gap, this paper for the first time systematically investigated the relationship between adversarially robust generalization and architectural design. Inparticular, we compr…
Embry-Riddle Aeronautical University（安柏瑞德航空大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Tennessee at Martin（田纳西大学马丁分校）；Bowling Green State University（鲍灵格林州立大学）；Purdue University（普渡大学）；Trier University of Applied Sciences（特里尔应用技术大学）
Deep Learning (DL) is being applied in various domains, especially in safety-critical applications such as autonomous driving. Consequently, it is of great significance to ensure the robustness of these methods and thus counteract uncertain behaviors caused by adversarial attacks. In this paper, we use gradient heatmaps to analyze the response characteristics of the VGG-16 model when the input images are mixed with adversarial noise and statistically similar Gaussian random noise. In particular, we compare the network response layer by layer to determine where errors occurred. Several interest…
Exploring Gradient-Guided Masked Language Model to Detect Textual Adversarial Attacks
School of Information and Communication Technology, Griffith University（格里菲斯大学信息与通信技术学院）；School of Computer Science, University of Technology Sydney（悉尼科技大学计算机科学学院）；College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；School of Cyber Science and Engineering, Huazhong University of Science and Technology（华中科技大学网络空间科学与工程学院）
The Pennsylvania State University（宾夕法尼亚州立大学）；Shanghai Jiao Tong University（上海交通大学）；Amplitude（Amplitude公司）
Deep neural networks (DNNs) have been broadly adopted in health risk prediction to provide healthcare diagnoses and treatments. To evaluate their robustness, existing research conducts adversarial attacks in the white/gray-box setting where model parameters are accessible. However, a more realistic black-box adversarial attack is ignored even though most real-world models are trained with private data and released as black-box services on the cloud. To fill this gap, we propose the first black-box adversarial attack method against health risk prediction models named MedAttacker to investigate …
Exploring adversarial robustness of JPEG AI: methodology, comparison and new methods
Lomonosov Moscow State University（莫斯科罗蒙诺索夫国立大学）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院信息学与管理研究中心可信人工智能研究中心）；MSU Institute for Artificial Intelligence（莫斯科国立大学人工智能研究所）；Innopolis University（Innopolis大学）
Beyond Crash: Hijacking Your Autonomous Vehicle for Fun and Profit
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
University of Florida（佛罗里达大学）；National Cheng Kung University（国立成功大学）；University of Central Florida（中佛罗里达大学）
Machine learning researchers have long noticed the phenomenon that the model training process will be more effective and efficient when the training samples are densely sampled around the underlying decision boundary. While this observation has already been widely applied in a range of machine learning security techniques, it lacks theoretical analyses of the correctness of the observation. To address this challenge, we first add particular perturbation to original training examples using adversarial attack methods so that the generated examples could lie approximately on the decision boundary…
Exploring the Effect of DNN Depth on Adversarial Attacks in Network Intrusion Detection Systems
Computer Engineering Carleton University Ottawa, Canada（计算机工程 卡尔顿大学渥太华加拿大）；School of Information Technology Carleton University Ottawa, Canada（信息科技学院 卡尔顿大学渥太华加拿大）
Various facial manipulation techniques have drawn serious public concerns in morality, security, and privacy. Although existing face forgery classifiers achieve promising performance on detecting fake images, these methods are vulnerable to adversarial examples with injected imperceptible perturbations on the pixels. Meanwhile, many face forgery detectors always utilize the frequency diversity between real and fake faces as a crucial clue. In this paper, instead of injecting adversarial perturbations into the spatial domain, we propose a frequency adversarial attack method against face forgery…
Integrated Simulation Framework for Adversarial Attacks on Autonomous Vehicles
Industrial Systems Institute, Athena Research Center, Patras Science Park, Greece（工业系统研究所，亚历山德罗斯研究中心，帕特拉科学公园，希腊）；AviSense.AI, Patras Science Park, Greece（AviSense.AI，帕特拉科学公园，希腊）；Dpt. of Informatics & Telecom., University of Ioannina, Arta, Greece（信息与电信系，伊奥安尼纳大学，阿塔，希腊）
Information Technology University (ITU)（信息技术大学（ITU））；Hamad Bin Khalifa University (HBKU)（哈马德本哈利法大学（HBKU））
Connected and autonomous vehicles (CAVs) will form the backbone of future next-generation intelligent transportation systems (ITS) providing travel comfort, road safety, along with a number of value-added services. Such a transformation---which will be fuelled by concomitant advances in technologies for machine learning (ML) and wireless communications---will enable a future vehicular ecosystem that is better featured and more efficient. However, there are lurking security problems related to the use of ML in such a critical setting where an incorrect ML decision may not only be a nuisance but…
The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspective
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Ruhr University Bochum（波鸿鲁尔大学）；Institute for AI in medicine (IKIM)（医学人工智能研究所）；University Hospital Essen（埃森大学医院）；Monash University（莫纳什大学）
RSC-SNN: Exploring the Trade-off Between Adversarial Robustness and Accuracy in Spiking Neural Networks via Randomized Smoothing Coding
Chongqing University（重庆大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Xi’an Jiaotong University（西安交通大学）；Software Security Technology Company Ltd（软件安全技术有限公司）
Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；University of Oxford（牛津大学）；Sun Yat-sen University（中山大学）
