This paper presents a safe reinforcement learning system for automated driving that benefits from multimodal future trajectory predictions. We propose a safety system that consists of two safety components: a heuristic safety and a learning-based safety. The heuristic safety module is based on common driving rules. On the other hand, the learning-based safety module is a data-driven safety rule that learns safety patterns from driving data. Specifically, it utilizes mixture density recurrent neural networks (MD-RNN) for multimodal future trajectory predictions to accelerate the learning progress. Our simulation results demonstrate that the proposed safety system outperforms previously reported results in terms of average reward and number of collisions.
We present a case study applying learning-based distributionally robust model predictive control to highway motion planning under stochastic uncertainty of the lane change behavior of surrounding road users. The dynamics of road users are modelled using Markov jump systems, in which the switching variable describes the desired lane of the vehicle under consideration and the continuous state describes the pose and velocity of the vehicles. We assume the switching probabilities of the underlying Markov chain to be unknown. As the vehicle is observed and thus, samples from the Markov chain are dr…
This paper presents a safe imitation learning approach for autonomous vehicle driving, with attention on real-life human driving data and experimental validation. In order to increase occupant's acceptance and gain drivers' trust, the autonomous driving function needs to provide a both safe and comfortable behavior such as risk-free and naturalistic driving. Our goal is to obtain such behavior via imitation learning of a planning policy from human driving data. In particular, we propose to incorporate barrier functions and smooth spline-based motion parametrization in the training loss functio…
West Virginia University（西弗吉尼亚大学）；Ford Greenfield Labs（福特Greenfield实验室）；University of Michigan（密歇根大学）；Ford Research and Innovation Center（福特研究与创新中心）
In this paper, we present a safe deep reinforcement learning system for automated driving. The proposed framework leverages merits of both rule-based and learning-based approaches for safety assurance. Our safety system consists of two modules namely handcrafted safety and dynamically-learned safety. The handcrafted safety module is a heuristic safety rule based on common driving practice that ensure a minimum relative gap to a traffic vehicle. On the other hand, the dynamically-learned safety module is a data-driven safety rule that learns safety patterns from driving data. Specifically, the …
The use of neural networks and reinforcement learning has become increasingly popular in autonomous vehicle control. However, the opaqueness of the resulting control policies presents a significant barrier to deploying neural network-based control in autonomous vehicles. In this paper, we present a reinforcement learning based approach to autonomous vehicle longitudinal control, where the rule-based safety cages provide enhanced safety for the vehicle as well as weak supervision to the reinforcement learning agent. By guiding the agent to meaningful states and actions, this weak supervision im…
With the advent of rapid globalization and the inter-border supply chain network, the reliability and efficiency of transportation systems have become even more critical. Indeed, trans-continental highways need particular attention due to their important role in sustaining globalization. In this context, intelligent transportation systems (ITS) can actively enhance the safety, mobility, productivity, and comfort of trans-continental highways. However, ITS efficiency depends greatly on the roads where they are deployed, on the availability of power and connectivity, and on the integration of fu…
Vehicles on highway on-ramps are one of the leading contributors to congestion. In this paper, we propose a prediction framework that predicts the longitudinal trajectories and lane changes (LCs) of vehicles on highway on-ramps and tapers. Specifically, our framework adopts a combination of prediction models that inputs a 4 seconds duration of a trajectory to output a forecast of the longitudinal trajectories and LCs up to 15 seconds ahead. Training and Validation based on next generation simulation (NGSIM) data show that the prediction power of the developed model and its accuracy outperforms…
School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；University of California at Berkeley（加州大学伯克利分校）
Humans make daily routine decisions based on their internal states in intricate interaction scenarios. This paper presents a probabilistically reconstructive learning approach to identify the internal states of multi-vehicle sequential interactions when merging at highway on-ramps. We treated the merging task's sequential decision as a dynamic, stochastic process and then integrated the internal states into an HMM-GMR model, a probabilistic combination of an extended Gaussian mixture regression (GMR) and hidden Markov models (HMM). We also developed a variant expectation-maximum (EM) algorithm…
In recent years, automotive technology has made a steady progress. In particular, Advanced Driver Assistance System (ADAS) has enabled many safety features in commercial vehicles, for instance, pedestrian detection, lane keeping assist, emergency automatic braking, etc. Although these features provide drivers with a safer operational environment, crashes still happen occasionally due to the complex road conditions and the unpredictable movement of road users including vehicles, pedestrians, bicyclists, and non-motorized vehicles. In this paper, we aim at predicting the possibilities of crashes…
Delft University of Technology（代尔夫特理工大学）；Beijing Institute of Technology（北京理工大学）；Technische Universität Dresden（德累斯顿工业大学）
Real-time safety systems are crucial components of intelligent vehicles. This paper introduces a prediction-based collision risk assessment approach on highways. Given a point mass vehicle dynamics system, a stochastic forward reachable set considering two-dimensional motion with vehicle state probability distributions is firstly established. We then develop an acceleration prediction model, which provides multi-modal probabilistic acceleration distributions to propagate vehicle states. The collision probability is calculated by summing up the probabilities of the states where two vehicles spa…
Legal Decision-making for Highway Automated Driving
school of mechanical engineering, Beijing Institute of Technology（北京理工大学机械工程学院）；Traffic Management Research Institute of the Ministry of Public Security（公安部交通管理科学研究所）；school of automotive and traffic engineering, Jiangsu University（江苏大学汽车与交通工程学院）
A risk-averse preview-based $Q$-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A risk assessment unit module is then presented that leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
In dense and dynamic scenarios, planning a safe and comfortable trajectory is full of challenges when traffic participants are driving at high speed. The classic graph search and sampling methods first perform path planning and then configure the corresponding speed, which lacks a strategy to deal with the high-speed obstacles. Decoupling optimization methods perform motion planning in the S-L and S-T domains respectively. These methods require a large free configuration space to plan the lane change trajectory. In dense dynamic scenes, it is easy to cause the failure of trajectory planning an…
Institute of Power Machinery and Vehicular Engineering, Faculty of Engineering, Zhejiang University（浙江大学工学部动力机械及车辆工程研究所）
Highway pilot assist has become the front line of competition in advanced driver assistance systems. The increasing requirements on safety and user acceptance are calling for personalization in the development process of such systems. Inspired by a finding on drivers' car-following preferences on lateral direction, a personalized highway pilot assist algorithm is proposed, which consists of an Intelligent Driver Model (IDM) based speed control model and a novel lane-keeping model considering the leading vehicle's lateral movement. A simulated driving experiment is conducted to analyse driver g…
On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards …
Automated vehicles require efficient and safe planning to maneuver in uncertain environments. Largely this uncertainty is caused by other traffic participants, e.g., surrounding vehicles. Future motion of surrounding vehicles is often difficult to predict. Whereas robust control approaches achieve safe, yet conservative motion planning for automated vehicles, Stochastic Model Predictive Control (SMPC) provides efficient planning in the presence of uncertainty. Probabilistic constraints are applied to ensure that the maximal risk remains below a predefined level. However, safety cannot be ensur…
Challenges related to automated driving are no longer focused on just the construction of such automated vehicles (AVs), but in assuring the safety of their operation. Recent advances in Level 3 and Level 4 autonomous driving have motivated more extensive study in safety guarantees of complicated AV maneuvers, which aligns with the goal of ISO 21448 (Safety of the Intended Functions, or SOTIF), i.e. minimizing unsafe scenarios both known and unknown, as well as Vision Zero -- eliminating highway fatalities by 2050. A majority of approaches used in providing safety guarantees for AV motion cont…
Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test
University of California, Berkeley（加州大学伯克利分校）；École des Ponts Paristech（巴黎高科路桥学院）；Vanderbilt University（范德堡大学）；Temple University（天普大学）；Rutgers University-Camden（罗格斯大学卡姆登分校）
Ensuring the safety of autonomous vehicles, given the uncertainty in sensing other road users, is an open problem. Moreover, separate safety specifications for perception and planning components raise how to assess the overall system safety. This work provides a probabilistic approach to calculate safety envelopes under perception uncertainty. The probabilistic envelope definition is based on a risk threshold. It limits the cumulative probability that the actual safety envelope in a fully observable environment is larger than an applied envelope and is solved using iterative worst-case analysi…
NVIDIA（英伟达）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）
Evaluating the safety of an autonomous vehicle (AV) depends on the behavior of surrounding agents which can be heavily influenced by factors such as environmental context and informally-defined driving etiquette. A key challenge is in determining a minimum set of assumptions on what constitutes reasonable foreseeable behaviors of other road users for the development of AV safety models and techniques. In this paper, we propose a data-driven AV safety design methodology that first learns ``reasonable'' behavioral assumptions from data, and then synthesizes an AV safety concept using these learn…
ITU Artificial Intelligence and Data Science Research Center and Department of Aeronautical Engineering, Istanbul Technical University（伊斯坦布尔技术大学人工智能与数据科学研究中心和航空工程系）；Eatron Technologies；Faculty of Mechatronics Engineering, Istanbul Technical University（伊斯坦布尔技术大学机电一体化工程学院）；Department of Computer Engineering, University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校计算机工程系）；Faculty of Aeronautics and Astronautics, Aerospace Engineering, Istanbul Technical University（伊斯坦布尔技术大学航空航天学院航空航天工程系）；Istanbul Technical University（伊斯坦布尔技术大学）；University of California, Santa Cruz（加利福尼亚大学圣克鲁兹分校）；Eatron Technologies, Istanbul, Turkey（土耳其伊斯坦布尔Eatron Technologies）；ITU Artificial Intelligence and Data Science Research Center（伊斯坦布尔技术大学人工智能与数据科学研究中心）；Department of Aeronautical Engineering, Istanbul Technical University（伊斯坦布尔技术大学航空工程系）
Autonomous lane changing is a critical feature for advanced autonomous driving systems, that involves several challenges such as uncertainty in other driver's behaviors and the trade-off between safety and agility. In this work, we develop a novel simulation environment that emulates these challenges and train a deep reinforcement learning agent that yields consistent performance in a variety of dynamic and uncertain traffic scenarios. Results show that the proposed data-driven approach performs significantly better in noisy environments compared to methods that rely solely on heuristics.
University of Michigan（密歇根大学）；Plus.ai Inc.（智加科技公司）；California Institute of Technology（加州理工学院）；Visteon Corporation（伟世通公司）
Accomplishing safe and efficient driving is one of the predominant challenges in the controller design of connected automated vehicles (CAVs). It is often more convenient to address these goals separately and integrate the resulting controllers. In this study, we propose a controller integration scheme to fuse performance-based controllers and safety-oriented controllers safely for the longitudinal motion of a CAV. The resulting structure is compatible with a large class of controllers, and offers flexibility to design each controller individually without affecting the performance of the other…
An Improved Deep Belief Network Model for Road Safety Analyses
Intelligent Transportation Systems Research Center（智能交通系统研究中心）；Wuhan University of Technology（武汉理工大学）；University of Waterloo（滑铁卢大学）；Department of Civil & Environmental Engineering（土木与环境工程系）；Department of Electrical & Computer Engineering（电气与计算机工程系）
Imitation learning has been widely used to learn control policies for autonomous driving based on pre-recorded data. However, imitation learning based policies have been shown to be susceptible to compounding errors when encountering states outside of the training distribution. Further, these agents have been demonstrated to be easily exploitable by adversarial road users aiming to create collisions. To overcome these shortcomings, we introduce Adversarial Mixture Density Networks (AMDN), which learns two distributions from separate datasets. The first is a distribution of safe actions learned…
Motion planning in environments with multiple agents is critical to many important autonomous applications such as autonomous vehicles and assistive robots. This paper considers the problem of motion planning, where the controlled agent shares the environment with multiple uncontrolled agents. First, a predictive model of the uncontrolled agents is trained to predict all possible trajectories within a short horizon based on the scenario. The prediction is then fed to a motion planning module based on model predictive control. We proved generalization bound for the predictive model using three …
A connected and automated vehicle safety metric determines the performance of a subject vehicle (SV) by analyzing the data involving the interactions among the SV and other dynamic road users and environmental features. When the data set contains only a finite set of samples collected from the naturalistic mixed-traffic driving environment, a metric is expected to generalize the safety assessment outcome from the observed finite samples to the unobserved cases by specifying in what domain the SV is expected to be safe and how safe the SV is, statistically, in that domain. However, to the best …
The steady increase in the number of vehicles operating on the highways continues to exacerbate congestion, accidents, energy consumption, and greenhouse gas emissions. Emerging mobility systems, e.g., connected and automated vehicles (CAVs), have the potential to directly address these issues and improve transportation network efficiency and safety. In this paper, we consider a highway merging scenario and propose a framework for coordinating CAVs such that stop-and-go driving is eliminated. We use a decentralized form of the actor-critic approach to deep reinforcement learning$-$multi-agent …
