Simulation data can be utilized to extend real-world driving data in order to cover edge cases, such as vehicle accidents. The importance of handling edge cases can be observed in the high societal costs in handling car accidents, as well as potential dangers to human drivers. In order to cover a wide and diverse range of all edge cases, we systemically parameterize and simulate the most common accident scenarios. By applying this data to autonomous driving models, we show that transfer learning on simulated data sets provide better generalization and collision avoidance, as compared to random initialization methods. Our results illustrate that information from a model trained on simulated data can be inferred to a model trained on real-world data, indicating the potential influence of simulation data in real world models and advancements in handling of anomalous driving scenarios.
National Institute of Technology Warangal（瓦朗加尔国家理工学院）
Numerous Deep Learning and sensor-based models have been developed to detect potential accidents with an autonomous vehicle. However, a self-driving car needs to be able to detect accidents between other vehicles in its path and take appropriate actions such as to slow down or stop and inform the concerned authorities. In this paper, we propose a novel support system for self-driving cars that detects vehicular accidents through a dashboard camera. The system leverages the Mask R-CNN framework for vehicle detection and a centroid tracking algorithm to track the detected vehicle. Additionally, …
Current end-to-end deep learning driving models have two problems: (1) Poor generalization ability of unobserved driving environment when diversity of training driving dataset is limited (2) Lack of accident explanation ability when driving models don't work as expected. To tackle these two problems, rooted on the believe that knowledge of associated easy task is benificial for addressing difficult task, we proposed a new driving model which is composed of perception module for \textit{see and think} and driving module for \textit{behave}, and trained it with multi-task perception-related basi…
Road accidents are estimated to be the ninth leading cause of death across all age groups globally. 1.25 million people die annually from road accidents and Africa has the highest rate of road fatalities [1]. Research shows that three out of five road accidents are caused by driver-related behavioral factors [2]. Self-driving technology has the potential of saving lives lost to these preventable road accidents. Africa accounts for the majority of road fatalities and as such would benefit immensely from this technology. However, financial constraints prevent viable experimentation and research …
Stanford University（斯坦福大学）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, these scenarios must be realistic and challenging, but not impossible to drive through safely. In this work, we introduce STRIVE, a method to automatically generate challenging scenarios that cause a given planner to produce undesirable behavior, like collisions. To maintain scenario plausibility, the key idea is to leverage a learned model of traffic motion in the form of a graph-based conditional VAE. Scenario generation is formulated as an optimization in the …
Real-time Accident Anticipation for Autonomous Driving Through Monocular Depth-Enhanced 3D Modeling
University of Macau（澳门大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；New York University（纽约大学）；Central South University（中南大学）；Beihang University（北京航空航天大学）；University of Hawaii（夏威夷大学）
Traffic accident anticipation aims to predict accidents from dashcam videos as early as possible, which is critical to safety-guaranteed self-driving systems. With cluttered traffic scenes and limited visual cues, it is of great challenge to predict how long there will be an accident from early observed frames. Most existing approaches are developed to learn features of accident-relevant agents for accident anticipation, while ignoring the features of their spatial and temporal relations. Besides, current deterministic deep neural networks could be overconfident in false predictions, leading t…
Autonomous systems require identifying the environment and it has a long way to go before putting it safely into practice. In autonomous driving systems, the detection of obstacles and traffic lights are of importance as well as lane tracking. In this study, an autonomous driving system is developed and tested in the experimental environment designed for this purpose. In this system, a model vehicle having a camera is used to trace the lanes and avoid obstacles to experimentally study autonomous driving behavior. Convolutional Neural Network models were trained for Lane tracking. For the vehic…
Indiana University（印第安纳大学）；University of Michigan（密歇根大学）
Recognizing abnormal events such as traffic violations and accidents in natural driving scenes is essential for successful autonomous driving and advanced driver assistance systems. However, most work on video anomaly detection suffers from two crucial drawbacks. First, they assume cameras are fixed and videos have static backgrounds, which is reasonable for surveillance applications but not for vehicle-mounted cameras. Second, they pose the problem as one-class classification, relying on arduously hand-labeled training datasets that limit recognition to anomaly categories that have been expli…
Learning physically grounded traffic accident reconstruction from public accident reports
State Key Laboratory of Internet of Things for Smart City（物联网智能城市国家重点实验室）；Department of Civil Engineering（土木工程系）；Department of Computer and Information Science（计算机与信息科学系）
Machine Learning-Based Vehicle Intention Trajectory Recognition and Prediction for Autonomous Driving
University of Southern California（南加州大学）；Virginia Tech（弗吉尼亚理工大学）；Miami University（迈阿密大学）；Northern Arizona University（北亚利桑那大学）；Northeastern University（东北大学）
Tshwane University of Technology（茨瓦内理工大学）；University of Botswana（博茨瓦纳大学）
Road region recognition is a main feature that is gaining increasing attention from intellectuals because it helps autonomous vehicle to achieve a successful navigation without accident. However, different techniques based on camera sensor have been used by various researchers and outstanding results have been achieved. Despite their success, environmental noise like shadow leads to inaccurate recognition of road region which eventually leads to accident for autonomous vehicle. In this research, we conducted an investigation on shadow and its effects, optimized the road region recognition syst…
Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tongji University（同济大学）；The Hong Kong University of Science and Technology（香港科技大学）
Independent Researcher（独立研究者）；Dalian University of Technology（大连理工大学）；Tsinghua University（清华大学）；University of Michigan（密歇根大学）
Although a typical autopilot system far surpasses humans in term of sensing accuracy, performance stability and response agility, such a system is still far behind humans in the wisdom of understanding an unfamiliar environment with creativity, adaptivity and resiliency. Current AD brains are basically expert systems featuring logical computations, which resemble the thinking flow of a left brain working at tactical level. A right brain is needed to upgrade the safety of automated driving vehicle onto next generation by making intuitive strategical judgements that can supervise the tactical ac…
Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；Department of Systems Design Engineering, University of Waterloo（滑铁卢大学系统设计工程系）；David Cheriton school of Computer science, University of Waterloo（滑铁卢大学戴维·谢里顿计算机学院）；University of Waterloo（滑铁卢大学）
This paper analyzes the robustness of deep learning models in autonomous driving applications and discusses the practical solutions to address that.
Domain-Enhanced Dual-Branch Model for Efficient and Interpretable Accident Anticipation
State Key Laboratory of Internet of Things for Smart City（物联网智能城市国家重点实验室）；University of Macau（澳门大学）；Department of Civil Engineering（土木工程系）；Department of Computer and Information Science（计算机与信息科学系）；College of Transportation Engineering（交通工程学院）
Motion Comfort Optimization for Autonomous Vehicles: Concepts, Methods, and Techniques
University of North Texas（北德克萨斯大学）；Fordham University（福特汉姆大学）；Qatar University（卡塔尔大学）；University of Nebraska at Kearney（内布拉斯加大学卡尼尔分校）；Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
University of Pennsylvania（宾夕法尼亚大学）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the $\textit{de facto}$ evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling method…
Our autonomous driving simulation lab produces a high-precision 3D model simulating the parking lot. However, the current model still has poor rendering quality in some aspects. In this work, we develop a system to improve the rendering of the model and evaluate the quality of the rendered model.
Unmanned vehicle technologies are an area of great interest in theory and practice today. These technologies have advanced considerably after the first applications have been implemented and cause a rapid change in human life. Autonomous vehicles are also a big part of these technologies. The most important action of a driver has to do is to follow the lanes on the way to the destination. By using image processing and artificial intelligence techniques, an autonomous vehicle can move successfully without a driver help. They can go from the initial point to the specified target by applying pre-…
ENSTA Paris（巴黎高等先进技术学院）；Institut Polytechnique de Paris（巴黎理工学院）；INRIA（法国国家信息与自动化研究所）；Université Paris-Saclay（巴黎萨克雷大学）；CEA（法国原子能和替代能源委员会）；List（李斯特研究所）
In recent years, we have witnessed increasingly high performance in the field of autonomous end-to-end driving. In particular, more and more research is being done on driving in urban environments, where the car has to follow high level commands to navigate. However, few evaluations are made on the ability of these agents to react in an unexpected situation. Specifically, no evaluations are conducted on the robustness of driving agents in the event of a bad high-level command. We propose here an evaluation method, namely a benchmark that allows to assess the robustness of an agent, and to appr…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Maryland at College Park（马里兰大学帕克分校）
Autonomous driving has gained significant advancements in recent years. However, obtaining a robust control policy for driving remains challenging as it requires training data from a variety of scenarios, including rare situations (e.g., accidents), an effective policy architecture, and an efficient learning mechanism. We propose ADAPS for producing robust control policies for autonomous vehicles. ADAPS consists of two simulation platforms in generating and analyzing accidents to automatically produce labeled training data, and a memory-enabled hierarchical control policy. Additionally, ADAPS …
In a mixed-traffic scenario where both autonomous vehicles and human-driving vehicles exist, a timely prediction of driving intentions of nearby human-driving vehicles is essential for the safe and efficient driving of an autonomous vehicle. In this paper, a driving intention prediction method based on Hidden Markov Model (HMM) is proposed for autonomous vehicles. HMMs representing different driving intentions are trained and tested with field collected data from a flyover. When training the models, either discrete or continuous characterization of the mobility features of vehicles is applied.…
Autonomous driving has been among the most popular and challenging topics in the past few years. On the road to achieving full autonomy, researchers have utilized various sensors, such as LiDAR, camera, Inertial Measurement Unit (IMU), and GPS, and developed intelligent algorithms for autonomous driving applications such as object detection, object segmentation, obstacle avoidance, and path planning. High-definition (HD) maps have drawn lots of attention in recent years. Because of the high precision and informative level of HD maps in localization, it has immediately become one of the critica…
Westview High School（韦斯特维尤高中）；Berkeley DeepDrive Center（伯克利深度驾驶中心）
Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and …
A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Forschungszentrum Informatik（弗劳恩霍夫信息技术研究中心）；Hochschule Esslingen（埃斯林根应用技术大学）；University of Michigan（密歇根大学）；Voxel51；Karlsruher Institut für Technologie（卡尔斯鲁厄理工学院）
In this paper, we investigate a predictive approach for collision risk assessment in autonomous and assisted driving. A deep predictive model is trained to anticipate imminent accidents from traditional video streams. In particular, the model learns to identify cues in RGB images that are predictive of hazardous upcoming situations. In contrast to previous work, our approach incorporates (a) temporal information during decision making, (b) multi-modal information about the environment, as well as the proprioceptive state and steering actions of the controlled vehicle, and (c) information about…
RGB cameras are one of the most relevant sensors for autonomous driving applications. It is undeniable that failures of vehicle cameras may compromise the autonomous driving task, possibly leading to unsafe behaviors when images that are subsequently processed by the driving system are altered. To support the definition of safe and robust vehicle architectures and intelligent systems, in this paper we define the failure modes of a vehicle camera, together with an analysis of effects and known mitigations. Further, we build a software library for the generation of the corresponding failed image…
To improve the security and robustness of autonomous driving models, this paper presents SMET, a scenariobased metamorphic testing tool for autonomous driving models. The metamorphic relationship is divided into three dimensions (time, space, and event) and demonstrates its effectiveness through case studies in two types of autonomous driving models with different outputs.Experimental results show that this tool can well detect potential defects of the autonomous driving model, and complex scenes are more effective than simple scenes.
Traffic accident anticipation aims to accurately and promptly predict the occurrence of a future accident from dashcam videos, which is vital for a safety-guaranteed self-driving system. To encourage an early and accurate decision, existing approaches typically focus on capturing the cues of spatial and temporal context before a future accident occurs. However, their decision-making lacks visual explanation and ignores the dynamic interaction with the environment. In this paper, we propose Deep ReInforced accident anticipation with Visual Explanation, named DRIVE. The method simulates both the…
PerceptIn（感知创新公司）；South China University of Technology（华南理工大学）；Northwestern University（西北大学）
The infrastructure-vehicle cooperative autonomous driving approach depends on the cooperation between intelligent roads and intelligent vehicles. This approach is not only safer but also more economical compared to the traditional on-vehicle-only autonomous driving approach. In this paper, we introduce our real-world deployment experiences of cooperative autonomous driving, and delve into the details of new challenges and opportunities. Specifically, based on our progress towards commercial deployment, we follow a three-stage development roadmap of the cooperative autonomous driving approach:i…
We describe the computing tasks involved in autonomous driving, examine existing autonomous driving computing platform implementations. To enable autonomous driving, the computing stack needs to simultaneously provide high performance, low power consumption, and low thermal dissipation, at low cost. We discuss possible approaches to design computing platforms that will meet these needs.
AVD2: Accident Video Diffusion for Accident Video Description
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Academy of Interdisciplinary Studies, the Hong Kong University of Science and Technology（香港科技大学跨学科研究学院）；College of Communication Engineering, Jilin University（吉林大学通信工程学院）；School of Cyber Science and Engineering, Nanjing University of Science and Technology（南京理工大学网络空间安全学院）；School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）；College of Foreign Language and Literature, Fudan University（复旦大学外国语言文学学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））；Lightwheel AI（轻舟智航（Lightwheel AI））
Autonomous vehicles (AVs) are promoted as a technology that will create a future with effortless driving and virtually no traffic accidents. AV companies claim that, when fully developed, the technology will eliminate 94% of all accidents that are caused by human error. These AVs will likely avoid the large number of crashes caused by impaired, distracted or reckless drivers. But there remains a significant proportion of crashes for which no driver is directly responsible. In particular, the absence of connectivity of an AV with its neighboring vehicles (V2V) and the infrastructure (I2V) leads…
Recent developments in autonomous driving technology have proven that map data may be used, not only for general routing purposes, but also for to enhance and complement common sensor data. This document reviews the most commonly used interfaces and formats at each step of a selfhealing map data chain.
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
Due to the complex and dynamic character of intersection scenarios, the autonomous driving strategy at intersections has been a difficult problem and a hot point in the research of intelligent transportation systems in recent years. This paper gives a brief summary of state-of-the-art autonomous driving strategies at intersections. Firstly, we enumerate and analyze common types of intersection scenarios, corresponding simulation platforms, as well as related datasets. Secondly, by reviewing previous studies, we have summarized characteristics of existing autonomous driving strategies and class…
Autonomous driving clouds provide essential services to support autonomous vehicles. Today these services include but not limited to distributed simulation tests for new algorithm deployment, offline deep learning model training, and High-Definition (HD) map generation. These services require infrastructure support including distributed computing, distributed storage, as well as heterogeneous computing. In this paper, we present the details of how we implement a unified autonomous driving cloud infrastructure, and how we support these services on top of this infrastructure.
Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance
Department of Civil & Environmental Engineering University of Utah（土木与环境工程系 犹他大学）；Zachry Department of Civil and Environmental Engineering Texas A&M University（扎克里系 土木与环境工程系 德克萨斯农工大学）；Department of Computer Science & Engineering Texas A&M University（计算机科学与工程系 德克萨斯农工大学）
