Deep neural networks (DNNs) are widely used in autonomous driving due to their high accuracy for perception, decision, and control. In safety-critical systems like autonomous driving, executing tasks like sensing and perception in real-time is vital to the vehicle's safety, which requires the application's execution time to be predictable. However, non-negligible time variations are observed in DNN inference. Current DNN inference studies either ignore the time variation issue or rely on the scheduler to handle it. None of the current work explains the root causes of DNN inference time variations. Understanding the time variations of the DNN inference becomes a fundamental challenge in real-time scheduling for autonomous driving. In this work, we analyze the time variation in DNN inference in fine granularity from six perspectives: data, I/O, model, runtime, hardware, and end-to-end perception system. Six insights are derived in understanding the time variations for DNN inference.
World Models for Autonomous Driving: An Initial Survey
University of Macau（澳门大学）；College of Transportation Engineering, Tongji University（同济大学交通运输工程学院）；Tsinghua University（清华大学）；Macau University of Science and Technology（澳门科技大学）；University of Hawaii at Manoa（夏威夷大学马诺阿分校）
Uncertainties in Deep Neural Network (DNN)-based perception and vehicle's motion pose challenges to the development of safe autonomous driving vehicles. In this paper, we propose a safe motion planning framework featuring the quantification and propagation of DNN-based perception uncertainties and motion uncertainties. Contributions of this work are twofold: (1) A Bayesian Deep Neural network model which detects 3D objects and quantitatively captures the associated aleatoric and epistemic uncertainties of DNNs; (2) An uncertainty-aware motion planning algorithm (PU-RRT) that accounts for uncer…
Recent developments in autonomous driving technology have proven that map data may be used, not only for general routing purposes, but also for to enhance and complement common sensor data. This document reviews the most commonly used interfaces and formats at each step of a selfhealing map data chain.
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
Due to the complex and dynamic character of intersection scenarios, the autonomous driving strategy at intersections has been a difficult problem and a hot point in the research of intelligent transportation systems in recent years. This paper gives a brief summary of state-of-the-art autonomous driving strategies at intersections. Firstly, we enumerate and analyze common types of intersection scenarios, corresponding simulation platforms, as well as related datasets. Secondly, by reviewing previous studies, we have summarized characteristics of existing autonomous driving strategies and class…
Enhancing Predictability of Multi-Tenant DNN Inference for Autonomous Vehicles' Perception
Department of Electrical Engineering and Computer Science, University of Michigan（电气工程与计算机科学系，密歇根大学）；Department of Computer Science and Engineering, Yonsei University（计算机科学与工程系，延世大学）；Department of Computer and Information Sciences, University of Delaware（计算机与信息科学系，特拉华大学）
Motion Comfort Optimization for Autonomous Vehicles: Concepts, Methods, and Techniques
University of North Texas（北德克萨斯大学）；Fordham University（福特汉姆大学）；Qatar University（卡塔尔大学）；University of Nebraska at Kearney（内布拉斯加大学卡尼尔分校）；Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios
Zhongguancun Laboratory（中关村实验室）；Beihang University（北京航空航天大学）；Institute of Data Space, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；A*STAR（新加坡科技研究局）
Owing to the merits of early safety and reliability guarantee, autonomous driving virtual testing has recently gains increasing attention compared with closed-loop testing in real scenarios. Although the availability and quality of autonomous driving datasets and toolsets are the premise to diagnose the autonomous driving system bottlenecks and improve the system performance, due to the diversity and privacy of the datasets and toolsets, collecting and featuring the perspective and quality of them become not only time-consuming but also increasingly challenging. This paper first proposes a Sys…
ENSTA Paris（巴黎高等先进技术学院）；Institut Polytechnique de Paris（巴黎理工学院）；INRIA（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；CEA（法国原子能和替代能源委员会）；List（李斯特研究所）
In recent years, we have witnessed increasingly high performance in the field of autonomous end-to-end driving. In particular, more and more research is being done on driving in urban environments, where the car has to follow high level commands to navigate. However, few evaluations are made on the ability of these agents to react in an unexpected situation. Specifically, no evaluations are conducted on the robustness of driving agents in the event of a bad high-level command. We propose here an evaluation method, namely a benchmark that allows to assess the robustness of an agent, and to appr…
In a mixed-traffic scenario where both autonomous vehicles and human-driving vehicles exist, a timely prediction of driving intentions of nearby human-driving vehicles is essential for the safe and efficient driving of an autonomous vehicle. In this paper, a driving intention prediction method based on Hidden Markov Model (HMM) is proposed for autonomous vehicles. HMMs representing different driving intentions are trained and tested with field collected data from a flyover. When training the models, either discrete or continuous characterization of the mobility features of vehicles is applied.…
This survey reviews explainability methods for vision-based self-driving systems trained with behavior cloning. The concept of explainability has several facets and the need for explainability is strong in driving, a safety-critical application. Gathering contributions from several research fields, namely computer vision, deep learning, autonomous driving, explainable AI (X-AI), this survey tackles several points. First, it discusses definitions, context, and motivation for gaining more interpretability and explainability from self-driving systems, as well as the challenges that are specific t…
We explore the complex design space of behaviour planning for autonomous driving. Design choices that successfully address one aspect of behaviour planning can critically constrain others. To aid the design process, in this work we decompose the design space with respect to important choices arising from the current state of the art approaches, and describe the resulting trade-offs. In doing this, we also identify interesting directions of future work.
Reinforcement learning has become one of the most trending subjects in the recent decade. It has seen applications in various fields such as robot manipulations, autonomous driving, path planning, computer gaming, etc. We accomplished three tasks during the course of this project. Firstly, we studied the Q-learning algorithm for tabular environments and applied it successfully to an OpenAi Gym environment, Taxi. Secondly, we gained an understanding of and implemented the deep Q-network algorithm for Cart-Pole environment. Thirdly, we also studied the application of reinforcement learning in au…
Westview High School（韦斯特维尤高中）；Berkeley DeepDrive Center（伯克利深度驾驶中心）
Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and …
Perceptin LLC（Perceptin有限责任公司）；Northeastern Univ（东北大学）；COCOPLE LLC（COCOPLE有限责任公司）；College of William and Mary（威廉玛丽学院）；North Carolina State University（北卡罗来纳州立大学）
Autonomous driving is of great interest in both research and industry. The high cost has been one of the major roadblocks that slow down the development and adoption of autonomous driving in practice. This paper, for the first-time, shows that it is possible to run level-4 (i.e., fully autonomous driving) software on a single off-the-shelf card (Jetson AGX Xavier) for less than $1k, an order of magnitude less than the state-of-the-art systems, while meeting all the requirements of latency. The success comes from the resolution of some important issues shared by existing practices through a ser…
With recent advances in learning algorithms and hardware development, autonomous cars have shown promise when operating in structured environments under good driving conditions. However, for complex, cluttered and unseen environments with high uncertainty, autonomous driving systems still frequently demonstrate erroneous or unexpected behaviors, that could lead to catastrophic outcomes. Autonomous vehicles should ideally adapt to driving conditions; while this can be achieved through multiple routes, it would be beneficial as a first step to be able to characterize Driveability in some quantif…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
In recent years, control under urban intersection scenarios becomes an emerging research topic. In such scenarios, the autonomous vehicle confronts complicated situations since it must deal with the interaction with social vehicles timely while obeying the traffic rules. Generally, the autonomous vehicle is supposed to avoid collisions while pursuing better efficiency. The existing work fails to provide a framework that emphasizes the integrity of the scenarios while being able to deploy and test reinforcement learning(RL) methods. Specifically, we propose a benchmark for training and testing …
The objective of the first CARLA autonomous driving challenge was to deploy autonomous driving systems to lead with complex traffic scenarios where all participants faced the same challenging traffic situations. According to the organizers, this competition emerges as a way to democratize and to accelerate the research and development of autonomous vehicles around the world using the CARLA simulator contributing to the development of the autonomous vehicle area. Therefore, this paper presents the architecture design for the navigation of an autonomous vehicle in a simulated urban environment t…
Based on the direct perception paradigm of autonomous driving, we investigate and modify the CNNs (convolutional neural networks) AlexNet and GoogLeNet that map an input image to few perception indicators (heading angle, distances to preceding cars, and distance to road centerline) for estimating driving affordances in highway traffic. We also design a controller with these indicators and the short-range sensor information of TORCS (the open racing car simulator) for driving simulated cars to avoid collisions. We collect a set of images from a TORCS camera in various driving scenarios, train t…
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
University of Surrey（萨里大学）；University of Warwick（华威大学）；Warwick Manufacturing Group（华威制造集团）
Autonomous driving decision-making is a challenging task due to the inherent complexity and uncertainty in traffic. For example, adjacent vehicles may change their lane or overtake at any time to pass a slow vehicle or to help traffic flow. Anticipating the intention of surrounding vehicles, estimating their future states and integrating them into the decision-making process of an automated vehicle can enhance the reliability of autonomous driving in complex driving scenarios. This paper proposes a Prediction-based Deep Reinforcement Learning (PDRL) decision-making model that considers the man…
To improve the security and robustness of autonomous driving models, this paper presents SMET, a scenariobased metamorphic testing tool for autonomous driving models. The metamorphic relationship is divided into three dimensions (time, space, and event) and demonstrates its effectiveness through case studies in two types of autonomous driving models with different outputs.Experimental results show that this tool can well detect potential defects of the autonomous driving model, and complex scenes are more effective than simple scenes.
South China University of Technology（华南理工大学）；PerceptIn Inc（PerceptIn公司）；University of Shanghai for Science and Technology（上海理工大学）；Université de Cergy-Pontoise（塞尔吉-蓬图瓦兹大学）；Clarkson University（克拉克森大学）；Wayne State University（韦恩州立大学）；University of California, Irvine（加利福尼亚大学欧文分校）
Autonomous driving is not one single technology but rather a complex system integrating many technologies, which means that teaching autonomous driving is a challenging task. Indeed, most existing autonomous driving classes focus on one of the technologies involved. This not only fails to provide a comprehensive coverage, but also sets a high entry barrier for students with different technology backgrounds. In this paper, we present a modular, integrated approach to teaching autonomous driving. Specifically, we organize the technologies used in autonomous driving into modules. This is describe…
PerceptIn（感知创新公司）；South China University of Technology（华南理工大学）；Northwestern University（西北大学）
The infrastructure-vehicle cooperative autonomous driving approach depends on the cooperation between intelligent roads and intelligent vehicles. This approach is not only safer but also more economical compared to the traditional on-vehicle-only autonomous driving approach. In this paper, we introduce our real-world deployment experiences of cooperative autonomous driving, and delve into the details of new challenges and opportunities. Specifically, based on our progress towards commercial deployment, we follow a three-stage development roadmap of the cooperative autonomous driving approach:i…
Autonomous systems require identifying the environment and it has a long way to go before putting it safely into practice. In autonomous driving systems, the detection of obstacles and traffic lights are of importance as well as lane tracking. In this study, an autonomous driving system is developed and tested in the experimental environment designed for this purpose. In this system, a model vehicle having a camera is used to trace the lanes and avoid obstacles to experimentally study autonomous driving behavior. Convolutional Neural Network models were trained for Lane tracking. For the vehic…
We describe the computing tasks involved in autonomous driving, examine existing autonomous driving computing platform implementations. To enable autonomous driving, the computing stack needs to simultaneously provide high performance, low power consumption, and low thermal dissipation, at low cost. We discuss possible approaches to design computing platforms that will meet these needs.
Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；Department of Systems Design Engineering, University of Waterloo（滑铁卢大学系统设计工程系）；David Cheriton school of Computer science, University of Waterloo（滑铁卢大学戴维·谢里顿计算机学院）；University of Waterloo（滑铁卢大学）
This paper analyzes the robustness of deep learning models in autonomous driving applications and discusses the practical solutions to address that.
Machine Learning-Based Vehicle Intention Trajectory Recognition and Prediction for Autonomous Driving
University of Southern California（南加州大学）；Virginia Tech（弗吉尼亚理工大学）；Miami University（迈阿密大学）；Northern Arizona University（北亚利桑那大学）；Northeastern University（东北大学）
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（华东师范大学）；Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）
Baidu Apollo Autonomous Driving（百度阿波罗自动驾驶）；University of Cambridge（剑桥大学）
We introduce ApolloRL, an open platform for research in reinforcement learning for autonomous driving. The platform provides a complete closed-loop pipeline with training, simulation, and evaluation components. It comes with 300 hours of real-world data in driving scenarios and popular baselines such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents. We elaborate in this paper on the architecture and the environment defined in the platform. In addition, we discuss the performance of the baseline agents in the ApolloRL environment.
Autonomous driving clouds provide essential services to support autonomous vehicles. Today these services include but not limited to distributed simulation tests for new algorithm deployment, offline deep learning model training, and High-Definition (HD) map generation. These services require infrastructure support including distributed computing, distributed storage, as well as heterogeneous computing. In this paper, we present the details of how we implement a unified autonomous driving cloud infrastructure, and how we support these services on top of this infrastructure.
In this paper, we continue our prior work on using imitation learning (IL) and model free reinforcement learning (RL) to learn driving policies for autonomous driving in urban scenarios, by introducing a model based RL method to drive the autonomous vehicle in the Carla urban driving simulator. Although IL and model free RL methods have been proved to be capable of solving lots of challenging tasks, including playing video games, robots, and, in our prior work, urban driving, the low sample efficiency of such methods greatly limits their applications on actual autonomous driving. In this work,…
This paper issues an integrated control system of self-driving autonomous vehicles based on the personal driving preference to provide personalized comfortable driving experience to autonomous vehicle users. We propose an Occupant's Preference Metric (OPM) which is defining a preferred lateral and longitudinal acceleration region with maximum allowable jerk for users. Moreover, we propose a vehicle controller based on control parameters enabling integrated lateral and longitudinal control via preference-aware maneuvering of autonomous vehicles. The proposed system not only provides the criteri…
Our autonomous driving simulation lab produces a high-precision 3D model simulating the parking lot. However, the current model still has poor rendering quality in some aspects. In this work, we develop a system to improve the rendering of the model and evaluate the quality of the rendered model.
Testing autonomous driving algorithms on real autonomous vehicles is extremely costly and many researchers and developers in the field cannot afford a real car and the corresponding sensors. Although several free and open-source autonomous driving stacks, such as Autoware and Apollo are available, choices of open-source simulators to use with them are limited. In this paper, we introduce the LGSVL Simulator which is a high fidelity simulator for autonomous driving. The simulator engine provides end-to-end, full-stack simulation which is ready to be hooked up to Autoware and Apollo. In addition…
University of Science and Technology of China（中国科学技术大学）；Carnegie Mellon University（卡内基梅隆大学）
In order to operate safely on the road, autonomous vehicles need not only to be able to identify objects in front of them, but also to be able to estimate the risk level of the object in front of the vehicle automatically. It is obvious that different objects have different levels of danger to autonomous vehicles. An evaluation system is needed to automatically determine the danger level of the object for the autonomous vehicle. It would be too subjective and incomplete if the system were completely defined by humans. Based on this, we propose a framework based on nonparametric Bayesian learni…
Since deep neural networks' resurgence, reinforcement learning has gradually strengthened and surpassed humans in many conventional games. However, it is not easy to copy these accomplishments to autonomous driving because state spaces are immensely complicated in the real world and action spaces are continuous and fine control is necessary. Besides, autonomous driving systems must also maintain their functionality regardless of the environment's complexity. The deep reinforcement learning domain (DRL) has become a robust learning framework to handle complex policies in high dimensional surrou…
