Although attention mechanisms have achieved considerable progress in Transformer-based architectures across various Artificial Intelligence (AI) domains, their inner workings remain to be explored. Existing explainable methods have different emphases but are rather one-sided. They primarily analyse the attention mechanisms or gradient-based attribution while neglecting the magnitudes of input feature values or the skip-connection module. Moreover, they inevitably bring spurious noisy pixel attributions unrelated to the model's decision, hindering humans' trust in the spotted visualization result. Hence, we propose an easy-to-implement but effective way to remedy this flaw: Smooth Noise Norm Attention (SNNA). We weigh the attention by the norm of the transformed value vector and guide the label-specific signal with the attention gradient, then randomly sample the input perturbations and average the corresponding gradients to produce noise-free attribution. Instead of evaluating the explanation method on the binary or multi-class classification tasks like in previous works, we explore the more complex multi-label classification scenario in this work, i.e., the driving action prediction task, and trained a model for it specifically. Both qualitative and quantitative evaluation results show the superiority of SNNA compared to other SOTA attention-based explainable methods in generating a clearer visual explanation map and ranking the input pixel importance.
NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3)（澳大利亚农村及偏远地区自动驾驶车辆ARC培训中心）；Université Gustave Eiffel（Gustave Eiffel大学）；Queensland University of Technology (QUT)（昆士兰科技大学）
In this era of active development of autonomous vehicles, it becomes crucial to provide driving systems with the capacity to explain their decisions. In this work, we focus on generating high-level driving explanations as the vehicle drives. We present BEEF, for BEhavior Explanation with Fusion, a deep architecture which explains the behavior of a trajectory prediction model. Supervised by annotations of human driving decisions justifications, BEEF learns to fuse features from multiple levels. Leveraging recent advances in the multi-modal fusion literature, BEEF is carefully designed to model …
University of California, Berkeley（加州大学伯克利分校）；MPI for Informatics（马克斯·普朗克信息学研究所）；University of Amsterdam（阿姆斯特丹大学）
Deep neural perception and control networks have become key components of self-driving vehicles. User acceptance is likely to benefit from easy-to-interpret textual explanations which allow end-users to understand what triggered a particular behavior. Explanations may be triggered by the neural controller, namely introspective explanations, or informed by the neural controller's output, namely rationalizations. We propose a new approach to introspective explanations which consists of two parts. First, we use a visual (spatial) attention model to train a convolutional network end-to-end from im…
Current end-to-end deep learning driving models have two problems: (1) Poor generalization ability of unobserved driving environment when diversity of training driving dataset is limited (2) Lack of accident explanation ability when driving models don't work as expected. To tackle these two problems, rooted on the believe that knowledge of associated easy task is benificial for addressing difficult task, we proposed a new driving model which is composed of perception module for \textit{see and think} and driving module for \textit{behave}, and trained it with multi-task perception-related basi…
Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
Institute for AI Industry Research (AIR)（人工智能产业研究院（AIR））；Xingjian College, Tsinghua University（清华大学行健书院）；OpenDriveLab, Shanghai AI Lab（上海人工智能实验室OpenDriveLab）；Mercedes-Benz Group China Ltd.（梅赛德斯-奔驰集团中国有限公司）
Integrated Gradients (IG) as well as its variants are well-known techniques for interpreting the decisions of deep neural networks. While IG-based approaches attain state-of-the-art performance, they often integrate noise into their explanation saliency maps, which reduce their interpretability. To minimize the noise, we examine the source of the noise analytically and propose a new approach to reduce the explanation noise based on our analytical findings. We propose the Important Direction Gradient Integration (IDGI) framework, which can be easily incorporated into any IG-based method that us…
Autonomous driving consists of a multitude of interacting modules, where each module must contend with errors from the others. Typically, the motion prediction module depends upon a robust tracking system to capture each agent's past movement. In this work, we systematically explore the importance of the tracking module for the motion prediction task and ultimately conclude that the overall motion prediction performance is highly sensitive to the tracking module's imperfections. We explicitly compare models that use tracking information to models that do not across multiple scenarios and condi…
Foreign substances on the road surface, such as rainwater or black ice, reduce the friction between the tire and the surface. The above situation will reduce the braking performance and make difficult to control the vehicle body posture. In that case, there is a possibility of property damage at least. In the worst case, personal damage will be occured. To avoid this problem, a road anomaly detection model is proposed based on vehicle driving noise. However, the prior proposal does not consider the extra noise, mixed with driving noise, and skipping calculations for moments without vehicle dri…
The complex driving environment brings great challenges to the visual perception of autonomous vehicles. It's essential to extract clear and explainable information from the complex road and traffic scenarios and offer clues to decision and control. However, the previous scene explanation had been implemented as a separate model. The black box model makes it difficult to interpret the driving environment. It cannot detect comprehensive textual information and requires a high computational load and time consumption. Thus, this study proposed a comprehensive and efficient textual explanation mod…
DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving
Tongji University（同济大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Macau（澳门大学）
Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-to-interpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers etc., can understand what triggered a particular behavior. Here we explore the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network's output (steering control). Our approach is two-stage. In the first stage, we use a visual attenti…
Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets
University of California San Diego（加利福尼亚大学圣迭戈分校）；Laboratory for Intelligent and Safe Automobiles（智能与安全汽车实验室）
Zhejiang University（浙江大学）；ETH Zürich（苏黎世联邦理工学院）；The Pennsylvania State University（宾夕法尼亚州立大学）
Understanding the decision process of neural networks is hard. One vital method for explanation is to attribute its decision to pivotal features. Although many algorithms are proposed, most of them solely improve the faithfulness to the model. However, the real environment contains many random noises, which may leads to great fluctuations in the explanations. More seriously, recent works show that explanation algorithms are vulnerable to adversarial attacks. All of these make the explanation hard to trust in real scenarios. To bridge this gap, we propose a model-agnostic method \emph{Median Te…
Purdue University（普渡大学）；Carnegie Mellon University（卡内基梅隆大学）；The Hong Kong Polytechnic University（香港理工大学）
In the last decade, deep learning (DL) approaches have been used successfully in computer vision (CV) applications. However, DL-based CV models are generally considered to be black boxes due to their lack of interpretability. This black box behavior has exacerbated user distrust and therefore has prevented widespread deployment DLCV models in autonomous driving tasks even though some of these models exhibit superiority over human performance. For this reason, it is essential to develop explainable DL models for autonomous driving task. Explainable DL models can not only boost user trust in aut…
Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Institute of Automotive Engineering, Technical University (TU) of Darmstadt（达姆施塔特工业大学汽车工程研究所）；School of Engineering and Physical Sciences, Heriot-Watt University（赫瑞瓦特大学工程与物理科学学院）
Road accident can be triggered by wet road because it decreases skid resistance. To prevent the road accident, detecting road surface abnomality is highly useful. In this paper, we propose the deep learning based cost-effective real-time anomaly detection architecture, naming with non-compression auto-encoder (NCAE). The proposed architecture can reflect forward and backward causality of time series information via convolutional operation. Moreover, the above architecture shows higher anomaly detection performance of published anomaly detection model via experiments. We conclude that NCAE as a…
Deep learning (DL) based computer vision (CV) models are generally considered as black boxes due to poor interpretability. This limitation impedes efficient diagnoses or predictions of system failure, thereby precluding the widespread deployment of DLCV models in safety-critical tasks such as autonomous driving. This study is motivated by the need to enhance the interpretability of DL model in autonomous driving and therefore proposes an explainable DL-based framework that generates textual descriptions of the driving environment and makes appropriate decisions based on the generated descripti…
Wet weather makes water film over the road and that film causes lower friction between tire and road surface. When a vehicle passes the low-friction road, the accident can occur up to 35% higher frequency than a normal condition road. In order to prevent accidents as above, identifying the road condition in real-time is essential. Thus, we propose a convolutional auto-encoder-based anomaly detection model for taking both less computational resources and achieving higher anomaly detection performance. The proposed model adopts a non-compression method rather than a conventional bottleneck struc…
We develop a novel framework to assess the risk of misperception in a traffic sign classification task in the presence of exogenous noise. We consider the problem in an autonomous driving setting, where visual input quality gradually improves due to improved resolution, and less noise since the distance to traffic signs decreases. Using the estimated perception statistics obtained using the standard classification algorithms, we aim to quantify the risk of misperception to mitigate the effects of imperfect visual observation. By exploring perception outputs, their expected high-level actions, …
Uber Advanced Technologies Group（优步先进技术集团）；Georgia Institute of Technology（佐治亚理工学院）
The capability to detect objects is a core part of autonomous driving. Due to sensor noise and incomplete data, perfectly detecting and localizing every object is infeasible. Therefore, it is important for a detector to provide the amount of uncertainty in each prediction. Providing the autonomous system with reliable uncertainties enables the vehicle to react differently based on the level of uncertainty. Previous work has estimated the uncertainty in a detection by predicting a probability distribution over object bounding boxes. In this work, we propose a method to improve the ability to le…
Georgia Institute of Technology（佐治亚理工学院）；KAUST（阿卜杜拉国王科技大学）
Performing driving behaviors based on causal reasoning is essential to ensure driving safety. In this work, we investigated how state-of-the-art 3D Convolutional Neural Networks (CNNs) perform on classifying driving behaviors based on causal reasoning. We proposed a perturbation-based visual explanation method to inspect the models' performance visually. By examining the video attention saliency, we found that existing models could not precisely capture the causes (e.g., traffic light) of the specific action (e.g., stopping). Therefore, the Temporal Reasoning Block (TRB) was proposed and intro…
