Safety is one of the most important development goals for highly automated driving (HAD) systems. This applies in particular to the perception function driven by Deep Neural Networks (DNNs). For these, large parts of the traditional safety processes and requirements are not fully applicable or sufficient. The aim of this paper is to present a framework for the description and mitigation of DNN insufficiencies and the derivation of relevant safety mechanisms to increase the safety of DNNs. To assess the effectiveness of these safety mechanisms, we present a categorization scheme for evaluation metrics.
Rain often poses inevitable threats to deep neural network (DNN) based perception systems, and a comprehensive investigation of the potential risks of the rain to DNNs is of great importance. However, it is rather difficult to collect or synthesize rainy images that can represent all rain situations that would possibly occur in the real world. To this end, in this paper, we start from a new perspective and propose to combine two totally different studies, i.e., rainy image synthesis and adversarial attack. We first present an adversarial rain attack, with which we could simulate various rain s…
Computer vision has made remarkable progress in recent years. Deep neural network (DNN) models optimized to identify objects in images exhibit unprecedented task-trained accuracy and, remarkably, some generalization ability: new visual problems can now be solved more easily based on previous learning. Biological vision (learned in life and through evolution) is also accurate and general-purpose. Is it possible that these different learning regimes converge to similar problem-dependent optimal computations? We therefore asked whether the human system-level computation of visual perception has D…
Enhancing Predictability of Multi-Tenant DNN Inference for Autonomous Vehicles' Perception
Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）；Department of Computer Science and Engineering, Yonsei University（计算机科学与工程系，延世大学）；Department of Computer and Information Sciences, University of Delaware（计算机与信息科学系，特拉华大学）
Divergences in Color Perception between Deep Neural Networks and Humans
Carnegie Observatories（卡内基天文台）；University of Southern California（南加州大学）；Stanford University（斯坦福大学）；EPFL（洛桑联邦理工学院）；University of Chicago（芝加哥大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Columbia University（哥伦比亚大学）；Psiphon Inc.（赛风公司）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Artificial intelligence (AI) systems power the world we live in. Deep neural networks (DNNs) are able to solve tasks in an ever-expanding landscape of scenarios, but our eagerness to apply these powerful models leads us to focus on their performance and deprioritises our ability to understand them. Current research in the field of explainable AI tries to bridge this gap by developing various perturbation or gradient-based explanation techniques. For images, these techniques fail to fully capture and convey the semantic information needed to elucidate why the model makes the predictions it does…
Valeo R&D Egypt（法雷奥研发埃及）；Valeo DAR Germany（法雷奥DAR德国）；Technische Universität Ilmenau（伊尔梅瑙工业大学）；Valeo Ireland（法雷奥爱尔兰）
Deep neural networks (DNNs) have accomplished impressive success in various applications, including autonomous driving perception tasks, in recent years. On the other hand, current deep neural networks are easily fooled by adversarial attacks. This vulnerability raises significant concerns, particularly in safety-critical applications. As a result, research into attacking and defending DNNs has gained much coverage. In this work, detailed adversarial attacks are applied on a diverse multi-task visual perception deep network across distance estimation, semantic segmentation, motion detection, a…
KBR（KBR公司）；NASA Ames（美国国家航空航天局艾姆斯研究中心）；Carnegie Mellon University（卡内基梅隆大学）；University of York（约克大学）；Boeing Research and Technology（波音研究与技术部）
Deep neural networks (DNNs) are increasingly used in safety-critical autonomous systems as perception components processing high-dimensional image data. Formal analysis of these systems is particularly challenging due to the complexity of the perception DNNs, the sensors (cameras), and the environment conditions. We present a case study applying formal probabilistic analysis techniques to an experimental autonomous system that guides airplanes on taxiways using a perception DNN. We address the above challenges by replacing the camera and the network with a compact probabilistic abstraction bui…
University of Science and Technology of China（中国科学技术大学）
Deep neural networks (DNNs) have shown great potential in non-reference image quality assessment (NR-IQA). However, the annotation of NR-IQA is labor-intensive and time-consuming, which severely limits their application especially for authentic images. To relieve the dependence on quality annotation, some works have applied unsupervised domain adaptation (UDA) to NR-IQA. However, the above methods ignore that the alignment space used in classification is sub-optimal, since the space is not elaborately designed for perception. To solve this challenge, we propose an effective perception-oriented…
Integrity Monitoring of 3D Object Detection in Automated Driving Systems using Raw Activation Patterns and Spatial Filtering
WMG, University of Warwick（华威大学WMG学院）；School of Electronics, Electrical Engineering and Computer Science (EEECS), Queen’s University of Belfast（贝尔法斯特女王大学电子、电气工程与计算机科学学院）
Kavli Institute for Systems Neuroscience（卡夫利系统神经科学研究所）；NTNU（挪威科技大学）；Max-Planck-Insitute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；UCL（伦敦大学学院）
Deep artificial neural networks (DNNs) trained through backpropagation provide effective models of the mammalian visual system, accurately capturing the hierarchy of neural responses through primary visual cortex to inferior temporal cortex (IT). However, the ability of these networks to explain representations in higher cortical areas is relatively lacking and considerably less well researched. For example, DNNs have been less successful as a model of the egocentric to allocentric transformation embodied by circuits in retrosplenial and posterior parietal cortex. We describe a novel scene per…
The recent success of Deep Neural Networks (DNNs) has revealed the significant capability of neural computing in many challenging applications. Although DNNs are derived from emulating biological neurons, there still exist doubts over whether or not DNNs are the final and best model to emulate the mechanism of human intelligence. In particular, there are two discrepancies between computational DNN models and the observed facts of biological neurons. First, human neurons are interconnected randomly, while DNNs need carefully-designed architectures to work properly. Second, human neurons usually…
Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception
Brown University（布朗大学）；Carney Institute for Brain Science, Brown University（布朗大学卡尼脑科学研究所）；Department of Computer Science, Brown University（布朗大学计算机科学系）；Institut de Recherche Technologique Saint-Exupéry（圣埃克苏佩里技术研究院）；Artificial and Natural Intelligence Toulouse Institute（图卢兹人工与自然智能研究所）
Massachusetts Institute of Technology（麻省理工学院）；Buckingham Browne & Nichols School（白金汉布朗与尼科尔斯学校）；Fujitsu Limited（富士通有限公司）
Symmetry is omnipresent in nature and perceived by the visual system of many species, as it facilitates detecting ecologically important classes of objects in our environment. Symmetry perception requires abstraction of long-range spatial dependencies between image regions, and its underlying neural mechanisms remain elusive. In this paper, we evaluate Deep Neural Network (DNN) architectures on the task of learning symmetry perception from examples. We demonstrate that feed-forward DNNs that excel at modelling human performance on object recognition tasks, are unable to acquire a general notio…
New York University（纽约大学）；NYU Langone Health（纽约大学朗格尼医学中心）；Warsaw University of Technology（华沙理工大学）
Deep neural networks (DNNs) show promise in image-based medical diagnosis, but cannot be fully trusted since their performance can be severely degraded by dataset shifts to which human perception remains invariant. If we can better understand the differences between human and machine perception, we can potentially characterize and mitigate this effect. We therefore propose a framework for comparing human and machine perception in medical diagnosis. The two are compared with respect to their sensitivity to the removal of clinically meaningful information, and to the regions of an image deemed m…
Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Wuppertal（伍珀塔尔大学）；Interdisciplinary Center for Machine Learning and Data Analytics（机器学习与数据分析跨学科中心）
Deep neural network (DNN) models have proven to be vulnerable to adversarial digital and physical attacks. In this paper, we propose a novel attack- and dataset-agnostic and real-time detector for both types of adversarial inputs to DNN-based perception systems. In particular, the proposed detector relies on the observation that adversarial images are sensitive to certain label-invariant transformations. Specifically, to determine if an image has been adversarially manipulated, the proposed detector checks if the output of the target classifier on a given input image changes significantly afte…
Adversarial examples have shown that albeit highly accurate, models learned by machines, differently from humans, have many weaknesses. However, humans' perception is also fundamentally different from machines, because we do not see the signals which arrive at the retina but a rather complex recreation of them. In this paper, we explore how machines could recreate the input as well as investigate the benefits of such an augmented perception. In this regard, we propose Perceptual Deep Neural Networks ($φ$DNN) which also recreate their own input before further processing. The concept is formaliz…
Deep neural networks (DNNs) have become remarkably successful in data prediction, and have even been used to predict future actions based on limited input. This raises the question: do these systems actually "understand" the event similar to humans? Here, we address this issue using videos taken from an accident situation in a driving simulation. In this situation, drivers had to choose between crashing into a suddenly-appeared obstacle or steering their car off a previously indicated cliff. We compared how well humans and a DNN predicted this decision as a function of time before the event. T…
Industrial Systems Institute, Athena Research Center（雅典娜研究中心工业系统研究所）；University of Patras（帕特雷大学）
Automotive Cyber-Physical Systems (ACPS) have attracted a significant amount of interest in the past few decades, while one of the most critical operations in these systems is the perception of the environment. Deep learning and, especially, the use of Deep Neural Networks (DNNs) provides impressive results in analyzing and understanding complex and dynamic scenes from visual data. The prediction horizons for those perception systems are very short and inference must often be performed in real time, stressing the need of transforming the original large pre-trained networks into new smaller mod…
Adversarial examples reveal the vulnerability and unexplained nature of neural networks. Studying the defense of adversarial examples is of considerable practical importance. Most adversarial examples that misclassify networks are often undetectable by humans. In this paper, we propose a defense model to train the classifier into a human-perception classification model with shape preference. The proposed model comprising a texture transfer network (TTN) and an auxiliary defense generative adversarial networks (GAN) is called Human-perception Auxiliary Defense GAN (HAD-GAN). The TTN is used to …
University of Wuppertal（伍珀塔尔大学）；Ruhr-University Bochum（波鸿鲁尔大学）；CARIAD SE；Volkswagen AG（大众汽车集团）；Center of Advanced Internet Studies（高级互联网研究中心）
Deep neural networks (DNN) have made impressive progress in the interpretation of image data, so that it is conceivable and to some degree realistic to use them in safety critical applications like automated driving. From an ethical standpoint, the AI algorithm should take into account the vulnerability of objects or subjects on the street that ranges from "not at all", e.g. the road itself, to "high vulnerability" of pedestrians. One way to take this into account is to define the cost of confusion of one semantic category with another and use cost-based decision rules for the interpretation o…
Intel Labs（英特尔实验室）；Technical University of Munich（慕尼黑工业大学）
Object detection neural network models need to perform reliably in highly dynamic and safety-critical environments like automated driving or robotics. Therefore, it is paramount to verify the robustness of the detection under unexpected hardware faults like soft errors that can impact a systems perception module. Standard metrics based on average precision produce model vulnerability estimates at the object level rather than at an image level. As we show in this paper, this does not provide an intuitive or representative indicator of the safety-related impact of silent data corruption caused b…
