Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation
fortiss GmbH（fortiss 有限公司）；Mentor Graphics（明导国际）；Technical University of Munich（慕尼黑工业大学）；German Aerospace Center (DLR)（德国航空航天中心）；Cognition Factory GmbH（认知工厂有限公司）
Annkathrin Krämmer, Christoph Schöller, Dhiraj Gulati, Venkatnarayanan Lakshminarasimhan, Franz Kurz, Dominik Rosenbaum, Claus Lenz, Alois Knoll
备注：
Accepted for publication in the Journal of Field Robotics
The environmental perception of an autonomous vehicle is limited by its physical sensor ranges and algorithmic performance, as well as by occlusions that degrade its understanding of an ongoing traffic situation. This not only poses a significant threat to safety and limits driving speeds, but it can also lead to inconvenient maneuvers. Intelligent Infrastructure Systems can help to alleviate these problems. An Intelligent Infrastructure System can fill in the gaps in a vehicle's perception and extend its field of view by providing additional detailed information about its surroundings, in the form of a digital model of the current traffic situation, i.e. a digital twin. However, detailed descriptions of such systems and working prototypes demonstrating their feasibility are scarce. In this paper, we propose a hardware and software architecture that enables such a reliable Intelligent Infrastructure System to be built. We have implemented this system in the real world and demonstrate its ability to create an accurate digital twin of an extended highway stretch, thus enhancing an autonomous vehicle's perception beyond the limits of its on-board sensors. Furthermore, we evaluate the accuracy and reliability of the digital twin by using aerial images and earth observation methods for generating ground truth data.
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Robert Bosch GmbH（罗伯特·博世有限公司）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；NEC Laboratories Europe GmbH（NEC欧洲实验室有限公司）；Leibniz University Hannover（汉诺威莱布尼茨大学）
In recent years, computer vision algorithms have become more powerful. However, current algorithms mainly share one limitation: They rely on directly visible objects. This is a significant drawback compared to human behavior, where visual cues caused by objects (e.g., shadows) are already used intuitively to retrieve information or anticipate occurring objects. While driving at night, this performance deficit becomes even more obvious: Humans already process the light artifacts caused by the headlamps of oncoming vehicles to estimate where they appear, whereas current object detection systems …
Mercedes-Benz AG（梅赛德斯-奔驰集团）；Princeton University（普林斯顿大学）；MESSRING Active Safety GmbH（MESSRING主动安全有限公司）
Current driver assistance systems and autonomous driving stacks are limited to well-defined environment conditions and geo fenced areas. To increase driving safety in adverse weather conditions, broadening the application spectrum of autonomous driving and driver assistance systems is necessary. In order to enable this development, reproducible benchmarking methods are required to quantify the expected distortions. In this publication, a testing methodology for disturbances from spray is presented. It introduces a novel lightweight and configurable spray setup alongside an evaluation scheme to…
Autonomous vehicles are being deployed with a spectrum of capability, extending from driver assistance features for the highway in personal vehicles (SAE Level 2+) to fully autonomous fleet ride sharing services operating in complex city environments (SAE Level 4+). This spectrum of autonomy often operates in different physical environments with different degrees of assumed driver in-the-loop oversight and hence have very different system and subsystem requirements. At the heart of SAE Level 2 to 5 systems is localization and mapping, which ranges from road determination for feature geofencing…
Crash data of autonomous vehicles (AV) or vehicles equipped with advanced driver assistance systems (ADAS) are the key information to understand the crash nature and to enhance the automation systems. However, most of the existing crash data sources are either limited by the sample size or suffer from missing or unverified data. To contribute to the AV safety research community, we introduce AVOID: an open AV crash dataset. Three types of vehicles are considered: Advanced Driving System (ADS) vehicles, Advanced Driver Assistance Systems (ADAS) vehicles, and low-speed autonomous shuttles. The c…
Amrita School of Engineering（阿姆里塔工程学院）；Amrita Vishwa Vidyapeetham（阿姆里塔维迪亚佩瑟姆大学）
Advanced driver assistance systems are advancing at a rapid pace and all major companies started investing in developing the autonomous vehicles. But the security and reliability is still uncertain and debatable. Imagine that a vehicle is compromised by the attackers and then what they can do. An attacker can control brake, accelerate and even steering which can lead to catastrophic consequences. This paper gives a very short and brief overview of most of the possible attacks on autonomous vehicle software and hardware and their potential implications.
Worcester Polytechnic Institute（伍斯特理工学院）；Toyota Motor North America（丰田汽车北美公司）
This study presents a novel approach for modeling and simulating human-vehicle interactions in order to examine the effects of automated driving systems (ADS) on driving performance and driver control workload. Existing driver-ADS interaction studies have relied on simulated or real-world human driver experiments that are limited in providing objective evaluation of the dynamic interactions and control workloads on the driver. Our approach leverages an integrated human model-based active driving system (HuMADS) to simulate the dynamic interaction between the driver model and the haptic-based A…
Texas A&M University（德克萨斯农工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
In this paper, an autonomous tethered Unmanned Aerial Vehicle (UAV) is developed into a visual assistant in a marsupial co-robots team, collaborating with a tele-operated Unmanned Ground Vehicle (UGV) for robot operations in unstructured or confined environments. These environments pose extreme challenges to the remote tele-operator due to the lack of sufficient situational awareness, mostly caused by the unstructuredness and confinement, stationary and limited field-of-view and lack of depth perception from the robot's onboard cameras. To overcome these problems, a secondary tele-operated rob…
Understanding human driving behavior is crucial to develop autonomous vehicles' algorithms. However, most low level automation, such as the one in advanced driving assistance systems (ADAS), is based on objective safety measures, which are not always aligned with what the drivers perceive as safe and their correspondent driving behavior. Finding the bridge between the subjective perception and objective safety measures has been analyzed in this paper focusing specifically on lane-change scenarios. Results showed statistically significant differences between what is perceived as safe by drivers…
Honda Research Institute（本田研究所）；School of Informatics, Computing and Engineering, Indiana University（印第安纳大学信息学、计算与工程学院）；Robotics Institute, University of Michigan（密歇根大学机器人研究所）
Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixel-level observations for future vehicle localization. We show that incorporating dense optical flow improves prediction resul…
University of Oxford（牛津大学）；FiveAI（FiveAI公司）；University of Colorado Boulder（科罗拉多大学博尔德分校）
Research into safety in autonomous and semi-autonomous vehicles has, so far, largely been focused on testing and validation through simulation. Due to the fact that failure of these autonomous systems is potentially life-endangering, formal methods arise as a complementary approach. This paper studies the application of formal methods to the verification of a human driver model built using the cognitive architecture ACT-R, and to the design of correct-by-construction Advanced Driver Assistance Systems (ADAS). The novelty lies in the integration of ACT-R in the formal analysis and an abstractio…
Human-computer interaction (HCI) is crucial for the safety of lives as autonomous vehicles (AVs) become commonplace. Yet, little effort has been put toward ensuring that AVs understand humans on the road. In this paper, we present GLADAS, a simulator-based research platform designed to teach AVs to understand pedestrian hand gestures. GLADAS supports the training, testing, and validation of deep learning-based self-driving car gesture recognition systems. We focus on gestures as they are a primordial (i.e, natural and common) way to interact with cars. To the best of our knowledge, GLADAS is t…
Institute of Mathematical and Computer Sciences (ICMC)（数学与计算机科学学院（ICMC））
One of the main paths towards the reduction of traffic accidents is the increase in vehicle safety through driver assistance systems or even systems with a complete level of autonomy. In these types of systems, tasks such as obstacle detection and segmentation, especially the Deep Learning-based ones, play a fundamental role in scene understanding for correct and safe navigation. Besides that, the wide variety of sensors in vehicles nowadays provides a rich set of alternatives for improvement in the robustness of perception in challenging situations, such as navigation under lighting and weath…
Modern driver assistance systems as well as autonomous vehicles take their decisions based on local maps of the environment. These maps include, for example, surrounding moving objects perceived by sensors as well as routes and navigation information. Current research in the field of environment mapping is concerned with two major challenges. The first one is the integration of information from different sources e.g. on-board sensors like radar, camera, ultrasound and lidar, offline map data or backend information. The second challenge comprises in finding an abstract representation of this ag…
GRASP Lab, University of Pennsylvania（宾夕法尼亚大学GRASP实验室）；Ford Autonomous Vehicles LLC（福特自动驾驶汽车有限责任公司）
Vehicle localization is essential for autonomous vehicle (AV) navigation and Advanced Driver Assistance Systems (ADAS). Accurate vehicle localization is often achieved via expensive inertial navigation systems or by employing compute-intensive vision processing (LiDAR/camera) to augment the low-cost and noisy inertial sensors. Here we have developed a framework for fusing the information obtained from a smart infrastructure node (ix-node) with the autonomous vehicles on-board localization engine to estimate the robust and accurate pose of the ego-vehicle even with cheap inertial sensors. A sma…
Aerial Assistance System for Automated Firefighting during Turntable Ladder Operations
Autonomous Intelligent Systems Group, Computer Science Institute VI – Intelligent Systems and Robotics（智能系统与机器人研究所）；Center for Robotics and Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany（机器人中心和拉马尔人工智能研究所，波恩大学，德国）；IDiAL – University of Applied Sciences and Arts Dortmund, Germany（IDiAL – 艾尔特大学应用科学与艺术学院，德国）；German Rescue Robotics Center (DRZ), Germany（德国救援机器人中心（DRZ），德国）；Institute for Fire Service and Rescue Technology – Fire Department of Dortmund, Germany（消防与救援技术研究所 – 多特蒙德消防部门，德国）
Johannes Kepler University Linz（约翰开普勒林茨大学）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Ural Federal University（乌拉尔联邦大学）
Rapid advancements in driver-assistance technology will lead to the integration of fully autonomous vehicles on our roads that will interact with other road users. To address the problem that driverless vehicles make interaction through eye contact impossible, we describe a framework for estimating the crossing intentions of pedestrians in order to reduce the uncertainty that the lack of eye contact between road users creates. The framework was deployed in a real vehicle and tested with three experimental cases that showed a variety of communication messages to pedestrians in a shared space sc…
Intrinsically, driving is a Markov Decision Process which suits well the reinforcement learning paradigm. In this paper, we propose a novel agent which learns to drive a vehicle without any human assistance. We use the concept of reinforcement learning and evolutionary strategies to train our agent in a 2D simulation environment. Our model's architecture goes beyond the World Model's by introducing difference images in the auto encoder. This novel involvement of difference images in the auto-encoder gives better representation of the latent space with respect to the motion of vehicle and helps…
Group control of connected and autonomous vehicles on automated highways is challenging for the advanced driver assistance systems (ADAS) and the automated driving systems (ADS). This paper investigates the differential game-based approach to autonomous convoy control with the aim of deployment on automated highways. Under the non-cooperative differential games, the coupled vehicles make their decisions independently while their states are interdependent. The receding horizon Nash equilibrium of the linear-quadratic differential game provides the convoy a distributed state-feedback control str…
MINES ParisTech（巴黎高科矿业学校）；PSL Research University（巴黎文理研究大学）；École des Ponts ParisTech（巴黎高科桥梁学院）
Before reaching full autonomy, vehicles will gradually be equipped with more and more advanced driver assistance systems (ADAS), effectively rendering them semi-autonomous. However, current ADAS technologies seem unable to handle complex traffic situations, notably when dealing with vehicles arriving from the sides, either at intersections or when merging on highways. The high rate of accidents in these settings prove that they constitute difficult driving situations. Moreover, intersections and merging lanes are often the source of important traffic congestion and, sometimes, deadlocks. In th…
Ford AV, LLC（福特自动驾驶汽车有限责任公司）；Ford Motor Company（福特汽车公司）；Xona Space Systems（Xona空间系统公司）
There is a growing need for vehicle positioning information to support Advanced Driver Assistance Systems (ADAS), Connectivity (V2X), and Autonomous Driving (AD) features. These range from a need for road determination ($<$5 meters), lane determination ($<$1.5 meters), and determining where the vehicle is within the lane ($<$0.3 meters). This paper presents the Ford Highway Driving RTK (Ford-HDR) dataset. This dataset includes nearly 30,000 km of data collected primarily on North American highways during a driving campaign designed to validate driver assistance features in 2018. This includes …
MINES ParisTech（巴黎高等矿业学院）；PSL Research University（巴黎文理研究大学）；École des Ponts ParisTech（巴黎高科桥路学校）
In order to drive safely and efficiently on public roads, autonomous vehicles will have to understand the intentions of surrounding vehicles, and adapt their own behavior accordingly. If experienced human drivers are generally good at inferring other vehicles' motion up to a few seconds in the future, most current Advanced Driving Assistance Systems (ADAS) are unable to perform such medium-term forecasts, and are usually limited to high-likelihood situations such as emergency braking. In this article, we present a first step towards consistent trajectory prediction by introducing a long short-…
Korea University（高丽大学）；Mofl Inc.（Mofl 公司）；Hallym University（翰林大学）；California State University, Long Beach（加州州立大学长滩分校）
Self-driving cars and autonomous driving research has been receiving considerable attention as major promising prospects in modern artificial intelligence applications. According to the evolution of advanced driver assistance system (ADAS), the design of self-driving vehicle and autonomous driving systems becomes complicated and safety-critical. In general, the intelligent system simultaneously and efficiently activates ADAS functions. Therefore, it is essential to consider reliable ADAS function coordination to control the driving system, safely. In order to deal with this issue, this paper p…
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）
The continuous advance of the automotive industry is leading to the emergence of more advanced driver assistance systems that enable the automation of certain tasks and that are undoubtedly aimed at achieving vehicles in which the driving task can be completely delegated. All these advances will bring changes in the paradigm of the automotive market, as is the case of insurance. For this reason, CESVIMAP and the Universidad Carlos III de Madrid are working on an Autonomous Testing pLatform for insurAnce reSearch (ATLAS) to study this technology and obtain first-hand knowledge about the respons…
OpenConvoy: Universal Platform for Real-World Testing of Cooperative Driving Systems
College of Engineering；Computer Science University of Central Florida Orlando, United States https://orcid.org/0009-0007-1826-373X；College of Electrical；Computer Engineering University of Central Florida Orlando, United States；Computer Science University of Central Florida Orlando, United States
University of Technology of Compiègne（贡比涅技术大学）；CNRS Heudiasyc UMR 7253（法国国家科学研究中心Heudiasyc UMR 7253实验室）
Taking into account the drivers' state is a major challenge for designing new advanced driver assistance systems. In this paper we present a driver assistance system strongly coupled to the user. DAARIA 1 stands for Driver Assistance by Augmented Reality for Intelligent Automobile. It is an augmented reality interface powered by several sensors. The detection has two goals: one is the position of obstacles and the quantification of the danger represented by them. The other is the driver's behavior. A suitable visualization metaphor allows the driver to perceive at any time the location of the …
CReSTIC, IUT de Troyes（特鲁瓦大学技术学院CReSTIC实验室）；Mines ParisTech, PSL Research University（巴黎高等矿业学院，PSL研究大学）；École polytechnique（巴黎综合理工学院）；AL.I.E.N.（AL.I.E.N.公司）；IFSTTAR（法国法国交通发展与网络科学研究院）；CNRS（法国国家科学研究中心）；Supélec（高等电力学院）；Université Paris-Sud（巴黎第十一大学）
A new model-free setting and the corresponding "intelligent" P and PD controllers are employed for the longitudinal and lateral motions of a vehicle. This new approach has been developed and used in order to ensure simultaneously a best profile tracking for the longitudinal and lateral behaviors. The longitudinal speed and the derivative of the lateral deviation, on one hand, the driving/braking torque and the steering angle, on the other hand, are respectively the output and the input variables. Let us emphasize that a "good" mathematical modeling, which is quite difficult, if not impossible …
LORIA-INRIA Lorraine（洛里亚-洛林因里亚研究所）；LAGIS-CNRS UMR 8146 Polytech'Lille（里尔理工大学LAGIS-CNRS联合研究所UMR 8146）
This paper presents a multi-sensor fusion strategy for a novel road-matching method designed to support real-time navigational features within advanced driving-assistance systems. Managing multihypotheses is a useful strategy for the road-matching problem. The multi-sensor fusion and multi-modal estimation are realized using Dynamical Bayesian Network. Experimental results, using data from Antilock Braking System (ABS) sensors, a differential Global Positioning System (GPS) receiver and an accurate digital roadmap, illustrate the performances of this approach, especially in ambiguous situation…
Institute of Industrial Science, the University of Tokyo（东京大学工业科学研究所）
Haptic guidance in a shared steering assistance system has drawn significant attention in intelligent vehicle fields, owing to its mutual communication ability for vehicle control. By exerting continuous torque on the steering wheel, both the driver and support system can share lateral control of the vehicle. However, current haptic guidance steering systems demonstrate some deficiencies in assisting lane changing. This study explored a new steering interaction method, including the design and evaluation of an intention-based haptic shared steering system. Such an intention-based method can su…
The Institute of Measurement, Control and Microtechnology at Ulm University investigates advanced driver assistance systems for decades and concentrates in large parts on autonomous driving. It is well known that motion planning is a key technology for autonomous driving. It is first and foremost responsible for the safety of the vehicle passengers as well as of all surrounding traffic participants. However, a further task consists in providing a smooth and comfortable driving behavior. In Ulm, we have the grateful opportunity to test our algorithms under real conditions in public traffic and …
Autonomous vehicles are the culmination of advances in many areas such as sensor technologies, artificial intelligence (AI), networking, and more. This paper will introduce the reader to the technologies that build autonomous vehicles. It will focus on open-source tools and libraries for autonomous vehicle development, making it cheaper and easier for developers and researchers to participate in the field. The topics covered are as follows. First, we will discuss the sensors used in autonomous vehicles and summarize their performance in different environments, costs, and unique features. Then …
Adaptive Cruise Control (ACC) is an Advanced Driver Assistance System (ADAS) that enables vehicle following with desired inter-vehicular distances. Cooperative Adaptive Cruise Control (CACC) is upgraded ACC that utilizes additional inter-vehicular wireless communication to share vehicle states such as acceleration to enable shorter gap following. Both ACC and CACC rely on range sensors such as radar to obtain the actual inter-vehicular distance for gap-keeping control. The range sensor may lose detection of the target, the preceding vehicle, on curvy roads or steep hills due to limited angle o…
Advanced Driver-Assistance Systems (ADAS) have been thriving and widely deployed in recent years. In general, these systems receive sensor data, compute driving decisions, and output control signals to the vehicles. To smooth out the uncertainties brought by sensor outputs, they usually leverage multi-sensor fusion (MSF) to fuse the sensor outputs and produce a more reliable understanding of the surroundings. However, MSF cannot completely eliminate the uncertainties since it lacks the knowledge about which sensor provides the most accurate data and how to optimally integrate the data provided…
Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Saarland University（萨尔大学）；AUDI AG（奥迪股份公司）
Due to the current developments towards autonomous driving and vehicle active safety, there is an increasing necessity for algorithms that are able to perform complex criticality predictions in real-time. Being able to process multi-object traffic scenarios aids the implementation of a variety of automotive applications such as driver assistance systems for collision prevention and mitigation as well as fall-back systems for autonomous vehicles. We present a fully model-based algorithm with a parallelizable architecture. The proposed algorithm can evaluate the criticality of complex, multi-mod…
School of Computer Science, National University of Ireland Galway, Ireland（爱尔兰国立高威大学计算机学院）
Autonomous robotics and artificial intelligence techniques can be used to support human personnel in the event of critical incidents. These incidents can pose great danger to human life. Some examples of such assistance include: multi-robot surveying of the scene; collection of sensor data and scene imagery, real-time risk assessment and analysis; object identification and anomaly detection; and retrieval of relevant supporting documentation such as standard operating procedures (SOPs). These incidents, although often rare, can involve chemical, biological, radiological/nuclear or explosive (C…
