Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrations
Shanghai Qizhi Institute（上海启智研究所）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息学院）；Mogo Auto Intelligence and Telematics Information Technology Co., Ltd（摩戈智能与 telemetry 信息技术有限公司）
An intelligent driving system should dynamically formulate appropriate driving strategies based on the current environment and vehicle status while ensuring system security and reliability. However, methods based on reinforcement learning and imitation learning often suffer from high sample complexity, poor generalization, and low safety. To address these challenges, this paper introduces an efficient and generalized end-to-end autonomous driving system (EGADS) for complex and varied scenarios. The RL agent in our EGADS combines variational inference with normalizing flows, which are independent of distribution assumptions. This combination allows the agent to capture historical information relevant to driving in latent space effectively, thereby significantly reducing sample complexity. Additionally, we enhance safety by formulating robust safety constraints and improve generalization and performance by integrating RL with expert demonstrations. Experimental results demonstrate that, compared to existing methods, EGADS significantly reduces sample complexity, greatly improves safety performance, and exhibits strong generalization capabilities in complex urban scenarios. Particularly, we contributed an expert dataset collected through human expert steering wheel control, specifically using the G29 steering wheel.
Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum
Seoul National University（首尔国立大学）；Artificial Intelligence Institute of Seoul National University (AIIS)（首尔国立大学人工智能研究所（AIIS））；Automation and Systems Research Institute (ASRI)（自动化与系统研究所（ASRI））
In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urb…
World Models for Autonomous Driving: An Initial Survey
University of Macau（澳门大学）；College of Transportation Engineering, Tongji University（同济大学交通运输工程学院）；Tsinghua University（清华大学）；Macau University of Science and Technology（澳门科技大学）；University of Hawaii at Manoa（夏威夷大学马诺阿分校）
In this paper, we develop a safe decision-making method for self-driving cars in a multi-lane, single-agent setting. The proposed approach utilizes deep reinforcement learning (RL) to achieve a high-level policy for safe tactical decision-making. We address two major challenges that arise solely in autonomous navigation. First, the proposed algorithm ensures that collisions never happen, and therefore accelerate the learning process. Second, the proposed algorithm takes into account the unobservable states in the environment. These states appear mainly due to the unpredictable behavior of othe…
Nanyang Technological University（南洋理工大学）；Energy Research Institute, Nanyang Technological University（南洋理工大学能源研究院）
Decision-making is critical for lane change in autonomous driving. Reinforcement learning (RL) algorithms aim to identify the values of behaviors in various situations and thus they become a promising pathway to address the decision-making problem. However, poor runtime safety hinders RL-based decision-making strategies from complex driving tasks in practice. To address this problem, human demonstrations are incorporated into the RL-based decision-making strategy in this paper. Decisions made by human subjects in a driving simulator are treated as safe demonstrations, which are stored into the…
US Army Research Laboratory（美国陆军研究实验室）；Texas A&M University（得克萨斯农工大学）；Columbia University（哥伦比亚大学）
This paper investigates how to utilize different forms of human interaction to safely train autonomous systems in real-time by learning from both human demonstrations and interventions. We implement two components of the Cycle-of-Learning for Autonomous Systems, which is our framework for combining multiple modalities of human interaction. The current effort employs human demonstrations to teach a desired behavior via imitation learning, then leverages intervention data to correct for undesired behaviors produced by the imitation learner to teach novel tasks to an autonomous agent safely, afte…
Recent developments in autonomous driving technology have proven that map data may be used, not only for general routing purposes, but also for to enhance and complement common sensor data. This document reviews the most commonly used interfaces and formats at each step of a selfhealing map data chain.
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
Due to the complex and dynamic character of intersection scenarios, the autonomous driving strategy at intersections has been a difficult problem and a hot point in the research of intelligent transportation systems in recent years. This paper gives a brief summary of state-of-the-art autonomous driving strategies at intersections. Firstly, we enumerate and analyze common types of intersection scenarios, corresponding simulation platforms, as well as related datasets. Secondly, by reviewing previous studies, we have summarized characteristics of existing autonomous driving strategies and class…
Motion Comfort Optimization for Autonomous Vehicles: Concepts, Methods, and Techniques
University of North Texas（北德克萨斯大学）；Fordham University（福特汉姆大学）；Qatar University（卡塔尔大学）；University of Nebraska at Kearney（内布拉斯加大学卡尼尔分校）；Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios
Zhongguancun Laboratory（中关村实验室）；Beihang University（北京航空航天大学）；Institute of Data Space, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）；Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；A*STAR（新加坡科技研究局）
Owing to the merits of early safety and reliability guarantee, autonomous driving virtual testing has recently gains increasing attention compared with closed-loop testing in real scenarios. Although the availability and quality of autonomous driving datasets and toolsets are the premise to diagnose the autonomous driving system bottlenecks and improve the system performance, due to the diversity and privacy of the datasets and toolsets, collecting and featuring the perspective and quality of them become not only time-consuming but also increasingly challenging. This paper first proposes a Sys…
We propose a new scheme to learn motion planning constraints from human driving trajectories. Behavioral and motion planning are the key components in an autonomous driving system. The behavioral planning is responsible for high-level decision making required to follow traffic rules and interact with other road participants. The motion planner role is to generate feasible, safe trajectories for a self-driving vehicle to follow. The trajectories are generated through an optimization scheme to optimize a cost function based on metrics related to smoothness, movability, and comfort, and subject t…
ENSTA Paris（巴黎高等先进技术学院）；Institut Polytechnique de Paris（巴黎理工学院）；INRIA（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；CEA（法国原子能和替代能源委员会）；List（李斯特研究所）
In recent years, we have witnessed increasingly high performance in the field of autonomous end-to-end driving. In particular, more and more research is being done on driving in urban environments, where the car has to follow high level commands to navigate. However, few evaluations are made on the ability of these agents to react in an unexpected situation. Specifically, no evaluations are conducted on the robustness of driving agents in the event of a bad high-level command. We propose here an evaluation method, namely a benchmark that allows to assess the robustness of an agent, and to appr…
In a mixed-traffic scenario where both autonomous vehicles and human-driving vehicles exist, a timely prediction of driving intentions of nearby human-driving vehicles is essential for the safe and efficient driving of an autonomous vehicle. In this paper, a driving intention prediction method based on Hidden Markov Model (HMM) is proposed for autonomous vehicles. HMMs representing different driving intentions are trained and tested with field collected data from a flyover. When training the models, either discrete or continuous characterization of the mobility features of vehicles is applied.…
This survey reviews explainability methods for vision-based self-driving systems trained with behavior cloning. The concept of explainability has several facets and the need for explainability is strong in driving, a safety-critical application. Gathering contributions from several research fields, namely computer vision, deep learning, autonomous driving, explainable AI (X-AI), this survey tackles several points. First, it discusses definitions, context, and motivation for gaining more interpretability and explainability from self-driving systems, as well as the challenges that are specific t…
We explore the complex design space of behaviour planning for autonomous driving. Design choices that successfully address one aspect of behaviour planning can critically constrain others. To aid the design process, in this work we decompose the design space with respect to important choices arising from the current state of the art approaches, and describe the resulting trade-offs. In doing this, we also identify interesting directions of future work.
Reinforcement learning has become one of the most trending subjects in the recent decade. It has seen applications in various fields such as robot manipulations, autonomous driving, path planning, computer gaming, etc. We accomplished three tasks during the course of this project. Firstly, we studied the Q-learning algorithm for tabular environments and applied it successfully to an OpenAi Gym environment, Taxi. Secondly, we gained an understanding of and implemented the deep Q-network algorithm for Cart-Pole environment. Thirdly, we also studied the application of reinforcement learning in au…
Technical University of Crete（克里特理工大学）；University of Malta（马耳他大学）
In this work the problem of path planning for an autonomous vehicle that moves on a freeway is considered. The most common approaches that are used to address this problem are based on optimal control methods, which make assumptions about the model of the environment and the system dynamics. On the contrary, this work proposes the development of a driving policy based on reinforcement learning. In this way, the proposed driving policy makes minimal or no assumptions about the environment, since a priori knowledge about the system dynamics is not required. Driving scenarios where the road is oc…
Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer
Institute of Trustworthy Embodied AI（可信具身人工智能研究所）；Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）；NVIDIA；University of Michigan（密歇根大学）；East China Normal University（华东师范大学）
Westview High School（韦斯特维尤高中）；Berkeley DeepDrive Center（伯克利深度驾驶中心）
Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and …
Perceptin LLC（Perceptin有限责任公司）；Northeastern Univ（东北大学）；COCOPLE LLC（COCOPLE有限责任公司）；College of William and Mary（威廉玛丽学院）；North Carolina State University（北卡罗来纳州立大学）
Autonomous driving is of great interest in both research and industry. The high cost has been one of the major roadblocks that slow down the development and adoption of autonomous driving in practice. This paper, for the first-time, shows that it is possible to run level-4 (i.e., fully autonomous driving) software on a single off-the-shelf card (Jetson AGX Xavier) for less than $1k, an order of magnitude less than the state-of-the-art systems, while meeting all the requirements of latency. The success comes from the resolution of some important issues shared by existing practices through a ser…
With recent advances in learning algorithms and hardware development, autonomous cars have shown promise when operating in structured environments under good driving conditions. However, for complex, cluttered and unseen environments with high uncertainty, autonomous driving systems still frequently demonstrate erroneous or unexpected behaviors, that could lead to catastrophic outcomes. Autonomous vehicles should ideally adapt to driving conditions; while this can be achieved through multiple routes, it would be beneficial as a first step to be able to characterize Driveability in some quantif…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
In recent years, control under urban intersection scenarios becomes an emerging research topic. In such scenarios, the autonomous vehicle confronts complicated situations since it must deal with the interaction with social vehicles timely while obeying the traffic rules. Generally, the autonomous vehicle is supposed to avoid collisions while pursuing better efficiency. The existing work fails to provide a framework that emphasizes the integrity of the scenarios while being able to deploy and test reinforcement learning(RL) methods. Specifically, we propose a benchmark for training and testing …
The objective of the first CARLA autonomous driving challenge was to deploy autonomous driving systems to lead with complex traffic scenarios where all participants faced the same challenging traffic situations. According to the organizers, this competition emerges as a way to democratize and to accelerate the research and development of autonomous vehicles around the world using the CARLA simulator contributing to the development of the autonomous vehicle area. Therefore, this paper presents the architecture design for the navigation of an autonomous vehicle in a simulated urban environment t…
Chongqing University（重庆大学）；University of Waterloo（滑铁卢大学）
Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning framework to transform the driving tasks in inter-section environments. The driving missions at the un-signalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The dec…
To improve the security and robustness of autonomous driving models, this paper presents SMET, a scenariobased metamorphic testing tool for autonomous driving models. The metamorphic relationship is divided into three dimensions (time, space, and event) and demonstrates its effectiveness through case studies in two types of autonomous driving models with different outputs.Experimental results show that this tool can well detect potential defects of the autonomous driving model, and complex scenes are more effective than simple scenes.
Based on the direct perception paradigm of autonomous driving, we investigate and modify the CNNs (convolutional neural networks) AlexNet and GoogLeNet that map an input image to few perception indicators (heading angle, distances to preceding cars, and distance to road centerline) for estimating driving affordances in highway traffic. We also design a controller with these indicators and the short-range sensor information of TORCS (the open racing car simulator) for driving simulated cars to avoid collisions. We collect a set of images from a TORCS camera in various driving scenarios, train t…
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
University of Surrey（萨里大学）；University of Warwick（华威大学）；Warwick Manufacturing Group（华威制造集团）
Autonomous driving decision-making is a challenging task due to the inherent complexity and uncertainty in traffic. For example, adjacent vehicles may change their lane or overtake at any time to pass a slow vehicle or to help traffic flow. Anticipating the intention of surrounding vehicles, estimating their future states and integrating them into the decision-making process of an automated vehicle can enhance the reliability of autonomous driving in complex driving scenarios. This paper proposes a Prediction-based Deep Reinforcement Learning (PDRL) decision-making model that considers the man…
PerceptIn（感知创新公司）；South China University of Technology（华南理工大学）；Northwestern University（西北大学）
The infrastructure-vehicle cooperative autonomous driving approach depends on the cooperation between intelligent roads and intelligent vehicles. This approach is not only safer but also more economical compared to the traditional on-vehicle-only autonomous driving approach. In this paper, we introduce our real-world deployment experiences of cooperative autonomous driving, and delve into the details of new challenges and opportunities. Specifically, based on our progress towards commercial deployment, we follow a three-stage development roadmap of the cooperative autonomous driving approach:i…
Autonomous Driving Small-Scale Cars: A Survey of Recent Development
Chair of Econometrics and Statistics, esp. in the Transport Sector, Technische Universität Dresden（经济计量学与统计学教授席位（侧重交通领域））；Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig（可扩展数据分析与人工智能中心（ScaDS.AI）德累斯顿/莱比锡）；Barkhausen Institut gGmbH（巴克豪森研究所）
Deep reinforcement learning is actively used for training autonomous car policies in a simulated driving environment. Due to the large availability of various reinforcement learning algorithms and the lack of their systematic comparison across different driving scenarios, we are unsure of which ones are more effective for training autonomous car software in single-agent as well as multi-agent driving environments. A benchmarking framework for the comparison of deep reinforcement learning in a vision-based autonomous driving will open up the possibilities for training better autonomous car driv…
South China University of Technology（华南理工大学）；PerceptIn Inc（PerceptIn公司）；University of Shanghai for Science and Technology（上海理工大学）；Université de Cergy-Pontoise（塞尔吉-蓬图瓦兹大学）；Clarkson University（克拉克森大学）；Wayne State University（韦恩州立大学）；University of California, Irvine（加利福尼亚大学欧文分校）
Autonomous driving is not one single technology but rather a complex system integrating many technologies, which means that teaching autonomous driving is a challenging task. Indeed, most existing autonomous driving classes focus on one of the technologies involved. This not only fails to provide a comprehensive coverage, but also sets a high entry barrier for students with different technology backgrounds. In this paper, we present a modular, integrated approach to teaching autonomous driving. Specifically, we organize the technologies used in autonomous driving into modules. This is describe…
We describe the computing tasks involved in autonomous driving, examine existing autonomous driving computing platform implementations. To enable autonomous driving, the computing stack needs to simultaneously provide high performance, low power consumption, and low thermal dissipation, at low cost. We discuss possible approaches to design computing platforms that will meet these needs.
Autonomous systems require identifying the environment and it has a long way to go before putting it safely into practice. In autonomous driving systems, the detection of obstacles and traffic lights are of importance as well as lane tracking. In this study, an autonomous driving system is developed and tested in the experimental environment designed for this purpose. In this system, a model vehicle having a camera is used to trace the lanes and avoid obstacles to experimentally study autonomous driving behavior. Convolutional Neural Network models were trained for Lane tracking. For the vehic…
Autonomous cars are well known for being vulnerable to adversarial attacks that can compromise the safety of the car and pose danger to other road users. To effectively defend against adversaries, it is required to not only test autonomous cars for finding driving errors but to improve the robustness of the cars to these errors. To this end, in this paper, we propose a two-step methodology for autonomous cars that consists of (i) finding failure states in autonomous cars by training the adversarial driving agent, and (ii) improving the robustness of autonomous cars by retraining them with effe…
Korea University（高丽大学）；Seoul National University（首尔国立大学）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）；ASRI, Seoul National University（首尔国立大学先进科学研究中心）；Department of Mechanical Engineering, Seoul National University（首尔国立大学机械工程系）；Department of Computer Science and Engineering, Seoul National University（首尔国立大学计算机科学与工程系）
Designing or learning an autonomous driving policy is undoubtedly a challenging task as the policy has to maintain its safety in all corner cases. In order to secure safety in autonomous driving, the ability to detect hazardous situations, which can be seen as an out-of-distribution (OOD) detection problem, becomes crucial. However, most conventional datasets only provide expert driving demonstrations, although some non-expert or uncommon driving behavior data are needed to implement a safety guaranteed autonomous driving platform. To this end, we present a novel dataset called the R3 Driving …
Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；Department of Systems Design Engineering, University of Waterloo（滑铁卢大学系统设计工程系）；David Cheriton school of Computer science, University of Waterloo（滑铁卢大学戴维·谢里顿计算机学院）；University of Waterloo（滑铁卢大学）
This paper analyzes the robustness of deep learning models in autonomous driving applications and discusses the practical solutions to address that.
Machine Learning-Based Vehicle Intention Trajectory Recognition and Prediction for Autonomous Driving
University of Southern California（南加州大学）；Virginia Tech（弗吉尼亚理工大学）；Miami University（迈阿密大学）；Northern Arizona University（北亚利桑那大学）；Northeastern University（东北大学）
Learning-based approaches, such as reinforcement learning (RL) and imitation learning (IL), have indicated superiority over rule-based approaches in complex urban autonomous driving environments, showing great potential to make intelligent decisions. However, current RL and IL approaches still have their own drawbacks, such as low data efficiency for RL and poor generalization capability for IL. In light of this, this paper proposes a novel learning-based method that combines deep reinforcement learning and imitation learning from expert demonstrations, which is applied to longitudinal vehicle…
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（华东师范大学）；Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）
Baidu Apollo Autonomous Driving（百度阿波罗自动驾驶）；University of Cambridge（剑桥大学）
We introduce ApolloRL, an open platform for research in reinforcement learning for autonomous driving. The platform provides a complete closed-loop pipeline with training, simulation, and evaluation components. It comes with 300 hours of real-world data in driving scenarios and popular baselines such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents. We elaborate in this paper on the architecture and the environment defined in the platform. In addition, we discuss the performance of the baseline agents in the ApolloRL environment.
