It is known that deep neural networks (DNNs) classify an input image by paying particular attention to certain specific pixels; a graphical representation of the magnitude of attention to each pixel is called a saliency-map. Saliency-maps are used to check the validity of the classification decision basis, e.g., it is not a valid basis for classification if a DNN pays more attention to the background rather than the subject of an image. Semantic perturbations can significantly change the saliency-map. In this work, we propose the first verification method for attention robustness, i.e., the local robustness of the changes in the saliency-map against combinations of semantic perturbations. Specifically, our method determines the range of the perturbation parameters (e.g., the brightness change) that maintains the difference between the actual saliency-map change and the expected saliency-map change below a given threshold value. Our method is based on activation region traversals, focusing on the outermost robust boundary for scalability on larger DNNs. Experimental results demonstrate that our method can show the extent to which DNNs can classify with the same basis regardless of semantic perturbations and report on performance and performance factors of activation region traversals.
Fraunhofer Institute for Integrated Circuits（弗劳恩霍夫集成电路研究所）；University of Bamberg（班贝格大学）
Neural networks with high performance can still be biased towards non-relevant features. However, reliability and robustness is especially important for high-risk fields such as clinical pain treatment. We therefore propose a verification pipeline, which consists of three steps. First, we classify facial expressions with a neural network. Next, we apply layer-wise relevance propagation to create pixel-based explanations. Finally, we quantify these visual explanations based on a bounding-box method with respect to facial regions. Although our results show that the neural network achieves state-…
While current approaches for neural network training often aim at improving performance, less focus is put on training methods aiming at robustness towards varying noise conditions or directed attacks by adversarial examples. In this paper, we propose to improve robustness by a multi-task training, which extends supervised semantic segmentation by a self-supervised monocular depth estimation on unlabeled videos. This additional task is only performed during training to improve the semantic segmentation model's robustness at test time under several input perturbations. Moreover, we even find th…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Center for Research and Formation in Artificial Intelligence Universidad de los Andes（安第斯大学人工智能研究与培训中心）；Facebook Reality Labs（脸书现实实验室）
Deep Neural Networks (DNNs) lack robustness against imperceptible perturbations to their input. Face Recognition Models (FRMs) based on DNNs inherit this vulnerability. We propose a methodology for assessing and characterizing the robustness of FRMs against semantic perturbations to their input. Our methodology causes FRMs to malfunction by designing adversarial attacks that search for identity-preserving modifications to faces. In particular, given a face, our attacks find identity-preserving variants of the face such that an FRM fails to recognize the images belonging to the same identity. W…
National Tsing Hua University（国立清华大学）；The Chinese University of Hong Kong（香港中文大学）；Columbia University（哥伦比亚大学）；IBM Research（IBM研究院）
Model robustness against adversarial examples of single perturbation type such as the $\ell_{p}$-norm has been widely studied, yet its generalization to more realistic scenarios involving multiple semantic perturbations and their composition remains largely unexplored. In this paper, we first propose a novel method for generating composite adversarial examples. Our method can find the optimal attack composition by utilizing component-wise projected gradient descent and automatic attack-order scheduling. We then propose generalized adversarial training (GAT) to extend model robustness from $\el…
On the Adversarial Robustness of Remote Sensing Semantic Change Detection
Linköping University（林雪平大学）；Helmholtz Institute Freiberg for Resource Technology（弗赖贝格亥姆霍兹资源技术研究所）；Helmholtz-Zentrum Dresden-Rossendorf（德累斯顿-罗森多夫亥姆霍兹中心）
University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Tencent AI Lab（腾讯人工智能实验室）
Recent studies imply that deep neural networks are vulnerable to adversarial examples -- inputs with a slight but intentional perturbation are incorrectly classified by the network. Such vulnerability makes it risky for some security-related applications (e.g., semantic segmentation in autonomous cars) and triggers tremendous concerns on the model reliability. For the first time, we comprehensively evaluate the robustness of existing UDA methods and propose a robust UDA approach. It is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, wh…
Transferring Styles for Reduced Texture Bias and Improved Robustness in Semantic Segmentation Networks
Department of Computer Science Heinrich-Heine-University Düsseldorf（计算机科学系 汉诺威-海因希-恩斯特大学）；Department of Mathematics University of Wuppertal（数学系 威斯巴登大学）
Deep Neural Networks (DNNs) have demonstrated exceptional performance on most recognition tasks such as image classification and segmentation. However, they have also been shown to be vulnerable to adversarial examples. This phenomenon has recently attracted a lot of attention but it has not been extensively studied on multiple, large-scale datasets and structured prediction tasks such as semantic segmentation which often require more specialised networks with additional components such as CRFs, dilated convolutions, skip-connections and multiscale processing. In this paper, we present what to…
Australian Centre for Field Robotics（澳大利亚现场机器人中心）；University of Sydney（悉尼大学）
One of the fundamental challenges in the design of perception systems for autonomous vehicles is validating the performance of each algorithm under a comprehensive variety of operating conditions. In the case of vision-based semantic segmentation, there are known issues when encountering new scenarios that are sufficiently different to the training data. In addition, even small variations in environmental conditions such as illumination and precipitation can affect the classification performance of the segmentation model. Given the reliance on visual information, these effects often translate …
Semantic segmentation is a scene understanding task at the heart of safety-critical applications where robustness to corrupted inputs is essential. Implicit Background Estimation (IBE) has demonstrated to be a promising technique to improve the robustness to out-of-distribution inputs for semantic segmentation models for little to no cost. In this paper, we provide analysis comparing the structures learned as a result of optimization objectives that use Softmax, IBE, and Sigmoid in order to improve understanding their relationship to robustness. As a result of this analysis, we propose combini…
Visual Learning Lab（视觉学习实验室）；Heidelberg University (HCI/IWR)（海德堡大学 (HCI/IWR)）
When designing a semantic segmentation module for a practical application, such as autonomous driving, it is crucial to understand the robustness of the module with respect to a wide range of image corruptions. While there are recent robustness studies for full-image classification, we are the first to present an exhaustive study for semantic segmentation, based on the state-of-the-art model DeepLabv3+. To increase the realism of our study, we utilize almost 400,000 images generated from Cityscapes, PASCAL VOC 2012, and ADE20K. Based on the benchmark study, we gain several new insights. Firstl…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））
Despite the impressive performance of Deep Neural Networks (DNNs) on various vision tasks, they still exhibit erroneous high sensitivity toward semantic primitives (e.g. object pose). We propose a theoretically grounded analysis for DNN robustness in the semantic space. We qualitatively analyze different DNNs' semantic robustness by visualizing the DNN global behavior as semantic maps and observe interesting behavior of some DNNs. Since generating these semantic maps does not scale well with the dimensionality of the semantic space, we develop a bottom-up approach to detect robust regions of D…
Semantic Robustness Probing via Inpainting: An Interactive Tool for Safety-Critical Object Detection
Federal Institute for Occupational Safety and Health (BAuA)（联邦职业安全与健康研究所）；Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫研究所（制造工程与自动化IPA））
The vulnerability of deep neural networks to adversarial examples, which are crafted maliciously by modifying the inputs with imperceptible perturbations to misled the network produce incorrect outputs, reveals the lack of robustness and poses security concerns. Previous works study the adversarial robustness of image classifiers on image level and use all the pixel information in an image indiscriminately, lacking of exploration of regions with different semantic meanings in the pixel space of an image. In this work, we fill this gap and explore the pixel space of the adversarial image by pro…
Tsinghua University（清华大学）；RealAI（瑞莱智慧）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute（清华大学-中国移动通信集团有限公司联合研究院）；Research Institute of Tsinghua University in Shenzhen（深圳清华大学研究院）
Certified defenses such as randomized smoothing have shown promise towards building reliable machine learning systems against $\ell_p$-norm bounded attacks. However, existing methods are insufficient or unable to provably defend against semantic transformations, especially those without closed-form expressions (such as defocus blur and pixelate), which are more common in practice and often unrestricted. To fill up this gap, we propose generalized randomized smoothing (GSmooth), a unified theoretical framework for certifying robustness against general semantic transformations via a novel dimens…
General Lipschitz: Certified Robustness Against Resolvable Semantic Transformations via Transformation-Dependent Randomized Smoothing
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Artificial Intelligence Research Institute（人工智能研究所）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院信息学研究所可信人工智能研究中心）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Sber AI Lab（俄罗斯联邦储蓄银行人工智能实验室）
针对图像模糊、平移等可解析语义变换及其组合，本文提出 General Lipschitz 框架，通过分析平滑分类器关于变换参数的 Lipschitz 连续性获得鲁棒性认证，并在 ImageNet 上达到与最先进方法相当的性能。
We address the problem of verifying neural-based perception systems implemented by convolutional neural networks. We define a notion of local robustness based on affine and photometric transformations. We show the notion cannot be captured by previously employed notions of robustness. The method proposed is based on reachability analysis for feed-forward neural networks and relies on MILP encodings of both the CNNs and transformations under question. We present an implementation and discuss the experimental results obtained for a CNN trained from the MNIST data set.
Even though deep neural networks succeed on many different tasks including semantic segmentation, they lack on robustness against adversarial examples. To counteract this exploit, often adversarial training is used. However, it is known that adversarial training with weak adversarial attacks (e.g. using the Fast Gradient Method) does not improve the robustness against stronger attacks. Recent research shows that it is possible to increase the robustness of such single-step methods by choosing an appropriate step size during the training. Finding such a step size, without increasing the computa…
Seoul National University（首尔国立大学）；Seoul, Korea（韩国首尔（注意：根据要求，地址信息不应提取为机构，此处可能是误判，正确机构应为Seoul National University相关部分））
Adversarial training is a defense technique that improves adversarial robustness of a deep neural network (DNN) by including adversarial examples in the training data. In this paper, we identify an overlooked problem of adversarial training in that these adversarial examples often have different semantics than the original data, introducing unintended biases into the model. We hypothesize that such non-semantics-preserving (and resultingly ambiguous) adversarial data harm the robustness of the target models. To mitigate such unintended semantic changes of adversarial examples, we propose seman…
Neural networks achieve outstanding accuracy in classification and regression tasks. However, understanding their behavior still remains an open challenge that requires questions to be addressed on the robustness, explainability and reliability of predictions. We answer these questions by computing reachable sets of neural networks, i.e. sets of outputs resulting from continuous sets of inputs. We provide two efficient approaches that lead to over- and under-approximations of the reachable set. This principle is highly versatile, as we show. First, we use it to analyze and enhance the robustne…
Verification of Geometric Robustness of Neural Networks via Piecewise Linear Approximation and Lipschitz Optimisation
Imperial College London（帝国理工学院）；University of California San Diego（加利福尼亚大学圣迭戈分校）；Inria（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学院）；PSL University（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；University of Limassol（利马索尔大学）；Safe Intelligence
To improve the robustness to rain, we present a physically-based rain rendering pipeline for realistically inserting rain into clear weather images. Our rendering relies on a physical particle simulator, an estimation of the scene lighting and an accurate rain photometric modeling to augment images with arbitrary amount of realistic rain or fog. We validate our rendering with a user study, proving our rain is judged 40% more realistic that state-of-the-art. Using our generated weather augmented Kitti and Cityscapes dataset, we conduct a thorough evaluation of deep object detection and semantic…
