The ability to identify important objects in a complex and dynamic driving environment is essential for autonomous driving agents to make safe and efficient driving decisions. It also helps assistive driving systems decide when to alert drivers. We tackle object importance estimation in a data-driven fashion and introduce HOIST - Human-annotated Object Importance in Simulated Traffic. HOIST contains driving scenarios with human-annotated importance labels for vehicles and pedestrians. We additionally propose a novel approach that relies on counterfactual reasoning to estimate an object's importance. We generate counterfactual scenarios by modifying the motion of objects and ascribe importance based on how the modifications affect the ego vehicle's driving. Our approach outperforms strong baselines for the task of object importance estimation on HOIST. We also perform ablation studies to justify our design choices and show the significance of the different components of our proposed approach.
fortiss GmbH（fortiss有限公司）；Technische Universität München（慕尼黑工业大学）
Learning-based approaches, such as reinforcement and imitation learning are gaining popularity in decision-making for autonomous driving. However, learned policies often fail to generalize and cannot handle novel situations well. Asking and answering questions in the form of "Would a policy perform well if the other agents had behaved differently?" can shed light on whether a policy has seen similar situations during training and generalizes well. In this work, a counterfactual policy evaluation is introduced that makes use of counterfactual worlds - worlds in which the behaviors of others are…
C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving
College of Transportation, Tongji University（同济大学交通运输学院）；Department of Civil Engineering, Tsinghua University（清华大学土木工程系）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Department of Civil and Environmental Engineering, National University of Singapore（新加坡国立大学土木与环境工程系）
PerceptIn（感知创新公司）；South China University of Technology（华南理工大学）；Northwestern University（西北大学）
The infrastructure-vehicle cooperative autonomous driving approach depends on the cooperation between intelligent roads and intelligent vehicles. This approach is not only safer but also more economical compared to the traditional on-vehicle-only autonomous driving approach. In this paper, we introduce our real-world deployment experiences of cooperative autonomous driving, and delve into the details of new challenges and opportunities. Specifically, based on our progress towards commercial deployment, we follow a three-stage development roadmap of the cooperative autonomous driving approach:i…
TiEV is an autonomous driving platform implemented by Tongji University of China. The vehicle is drive-by-wire and is fully powered by electricity. We devised the software system of TiEV from scratch, which is capable of driving the vehicle autonomously in urban paths as well as on fast express roads. We describe our whole system, especially novel modules of probabilistic perception fusion, incremental mapping, the 1st and the 2nd planning and the overall safety concern. TiEV finished 2016 and 2017 Intelligent Vehicle Future Challenge of China held at Changshu. We show our experiences on the d…
With the development of autonomous driving, the improvement of autonomous driving technology for individual vehicles has reached the bottleneck. The advancement of vehicle-road cooperation autonomous driving technology can expand the vehicle's perception range, supplement the perception blind area and improve the perception accuracy, to promote the development of autonomous driving technology and achieve vehicle-road integration. This project mainly uses lidar to develop data fusion schemes to realize the sharing and combination of vehicle and road equipment data and achieve the detection and …
Large Multimodal Models for Embodied Intelligent Driving: The Next Frontier in Self-Driving?
School of Information and Electrical Engineering, Hebei University of Engineering（河北工程大学信息与电子工程学院）；School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Department of Electrical and Computer Engineering, Auburn University（阿肯色大学电气与计算机工程系）；Department of Electrical and Computer Engineering, University of Houston（休斯顿大学电气与计算机工程系）；Department of Computer Science and Engineering, Kyung Hee University（庆熙大学计算机科学与工程系）；Machine Learning Department, Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed人工智能大学机器学习系）
The autonomous car technology promises to replace human drivers with safer driving systems. But although autonomous cars can become safer than human drivers this is a long process that is going to be refined over time. Before these vehicles are deployed on urban roads a minimum safety level must be assured. Since the autonomous car technology is still under development there is no standard methodology to evaluate such systems. It is important to completely understand the technology that is being developed to design efficient means to evaluate it. In this paper we assume safety-critical systems…
Department of Computer Science, University of Memphis（孟菲斯大学计算机科学系）；University of Memphis（孟菲斯大学）
We present a novel adaptive cruise control (ACC) system namely SAINT-ACC: {S}afety-{A}ware {Int}elligent {ACC} system (SAINT-ACC) that is designed to achieve simultaneous optimization of traffic efficiency, driving safety, and driving comfort through dynamic adaptation of the inter-vehicle gap based on deep reinforcement learning (RL). A novel dual RL agent-based approach is developed to seek and adapt the optimal balance between traffic efficiency and driving safety/comfort by effectively controlling the driving safety model parameters and inter-vehicle gap based on macroscopic and microscopi…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Waytous Ltd.（Waytous有限公司）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；Hong Kong Baptist University（香港浸会大学）；Hong Kong Polytechnical University（香港理工大学）；Hunan University（湖南大学）；Cranfield University（克兰菲尔德大学）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
Interest in autonomous driving (AD) and intelligent vehicles (IVs) is growing at a rapid pace due to the convenience, safety, and economic benefits. Although a number of surveys have reviewed research achievements in this field, they are still limited in specific tasks, lack of systematic summary and research directions in the future. Here we propose a Survey of Surveys (SoS) for total technologies of AD and IVs that reviews the history, summarizes the milestones, and provides the perspectives, ethics, and future research directions. To our knowledge, this article is the first SoS with milesto…
University of Oxford（牛津大学）；Umeå University（于默奥大学）；University of Nottingham（诺丁汉大学）；Oxford Robotics Institute（牛津机器人研究所）
In commentary driving, drivers verbalise their observations, assessments and intentions. By speaking out their thoughts, both learning and expert drivers are able to create a better understanding and awareness of their surroundings. In the intelligent vehicle context, automated driving commentary can provide intelligible explanations about driving actions, thereby assisting a driver or an end-user during driving operations in challenging and safety-critical scenarios. In this paper, we conducted a field study in which we deployed a research vehicle in an urban environment to obtain data. While…
SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Driving
College of Transportation, Tongji University（同济大学交通运输学院）；Department of Civil Engineering, Tsinghua University（清华大学土木工程系）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Department of Civil and Environmental Engineering, National University of Singapore（新加坡国立大学土木与环境工程系）
Milestones in Autonomous Driving and Intelligent Vehicles Part II: Perception and Planning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Waytous Ltd.（Waytous有限公司）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；Hong Kong Baptist University（香港浸会大学）；Hunan University（湖南大学）；University of Cambridge（剑桥大学）；Xi’an Jiaotong University（西安交通大学）；Tsinghua University（清华大学）
SEIDM: A Safe and Efficient Intelligent Driver Model for Autonomous Driving Behavior
Department of Information and Communication Engineering, Tongji University（信息与通信工程系，同济大学）；Department of Electrical and Computer Engineering, Duke University（电气与计算机工程系，杜克大学）
Australian Center for Field Robotics（澳大利亚现场机器人中心）；the University of Sydney（悉尼大学）
Robust and persistent localisation is essential for ensuring the safe operation of autonomous vehicles. When operating in large and diverse urban driving environments, autonomous vehicles are frequently exposed to situations that violate the assumptions of algorithms, suffer from the failure of one or more sensors, or other events that lead to a loss of localisation. This paper proposes the use of a behavior tree framework that can monitor the performance of localisation health metrics and triggers intelligent responses such as sensor switching and loss recovery. The algorithm presented select…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Delaware（特拉华大学）；University of Chinese Academy of Sciences（中国科学院大学）
Infrastructure-Vehicle Cooperative Autonomous Driving (IVCAD) is a new paradigm of autonomous driving, which relies on the cooperation between intelligent roads and autonomous vehicles. This paradigm has been shown to be safer and more efficient compared to the on-vehicle-only autonomous driving paradigm. Our real-world deployment data indicates that the effectiveness of IVCAD is constrained by reliability and performance of commercial communication networks. This paper targets this exact problem, and proposes INTERNEURON, a middleware to achieve high communication reliability between intellig…
The ability to estimate human intentions and interact with human drivers intelligently is crucial for autonomous vehicles to successfully achieve their objectives. In this paper, we propose a game theoretic planning algorithm that models human opponents with an iterative reasoning framework and estimates human latent cognitive states through probabilistic inference and active learning. By modeling the interaction as a partially observable Markov decision process with adaptive state and action spaces, our algorithm is able to accomplish real-time lane changing tasks in a realistic driving simul…
Markov Regime-Switching Intelligent Driver Model for Interpretable Car-Following Behavior
Department of Civil Engineering, McGill University, Montreal, QC H3A 0C3, Canada；Laboratory for Information \& Decision Systems (LIDS), Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Integrating Multi-Modal Sensors: A Review of Fusion Techniques for Intelligent Vehicles
College of Engineering, Center for Environmental Research and Technology, University of California at Riverside（工程学院、环境研究与技术中心、加州大学河滨分校）；School of Transportation and Logistics, Southwest Jiaotong University（交通运输与物流学院、西南交通大学）
Iran University of Science and Technology（伊朗科技大学）；Amirkabir University of Technology（阿米卡比尔理工大学）；University of Calgary（卡尔加里大学）
A significant portion of driving hazards is caused by human error and disregard for local driving regulations; Consequently, an intelligent assistance system can be beneficial. This paper proposes a novel vision-based modular package to ensure drivers' safety by perceiving the environment. Each module is designed based on accuracy and inference time to deliver real-time performance. As a result, the proposed system can be implemented on a wide range of vehicles with minimum hardware requirements. Our modular package comprises four main sections: lane detection, object detection, segmentation, …
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
Due to the complex and dynamic character of intersection scenarios, the autonomous driving strategy at intersections has been a difficult problem and a hot point in the research of intelligent transportation systems in recent years. This paper gives a brief summary of state-of-the-art autonomous driving strategies at intersections. Firstly, we enumerate and analyze common types of intersection scenarios, corresponding simulation platforms, as well as related datasets. Secondly, by reviewing previous studies, we have summarized characteristics of existing autonomous driving strategies and class…
Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Waytous Ltd.（Waytous有限公司）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；Hong Kong Baptist University（香港浸会大学）；Hong Kong Polytechnical University（香港理工大学）；Cranfield University（克兰菲尔德大学）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
Global Center for Automotive Performance Simulation（全球汽车性能模拟中心）；Honda Research Institute, USA Inc.（美国本田研究院）；Virginia Tech Transportation Institute（弗吉尼亚理工交通研究所）
Contextual Robotics Institute, University of California, San Diego（加州大学圣地亚哥分校语境机器人研究所）
Calibration of sensors is fundamental to robust performance for intelligent vehicles. In natural environments, disturbances can easily challenge calibration. One possibility is to use natural objects of known shape to recalibrate sensors. An approach based on recognition of traffic signs, such as stop signs, and use of them for recalibration of cameras is presented. The approach is based on detection, geometry estimation, calibration, and recursive updating. Results from natural environments are presented that clearly show convergence and improved performance.
The three keys to autonomous driving are sensors, data integration, and 100% safety decisions. In the past, due to the high latency and low reliability of the network, many decisions had to be made locally in the vehicle. This puts high demands on the vehicle itself, which results in the dilatory commercialization of automatic driving. With the advent of 5G, these situations will be greatly improved. In this paper, we present the improvements that 5G technology brings to autonomous vehicles especially in terms of latency and reliability amongst the multitude of other factors. The paper analyze…
An important topic in the autonomous driving research is the development of maneuver planning systems. Vehicles have to interact and negotiate with each other so that optimal choices, in terms of time and safety, are taken. For this purpose, we present a maneuver planning module able to negotiate the entering in busy roundabouts. The proposed module is based on a neural network trained to predict when and how entering the roundabout throughout the whole duration of the maneuver. Our model is trained with a novel implementation of A3C, which we will call Delayed A3C (D-A3C), in a synthetic envi…
University of Auckland（奥克兰大学）；IIT Bhubaneswar（布巴内斯瓦尔印度理工学院）
Cyber-Physical Systems (CPSs) such as those found within autonomous vehicles are increasingly adopting Artificial Neural Network (ANN)-based controllers. To ensure the safety of these controllers, there is a spate of recent activity to formally verify the ANN-based designs. There are two challenges with these approaches: (1) The verification of such systems is difficult and time consuming. (2) These verified controllers are not able to adapt to frequent requirements changes, which are typical in situations like autonomous driving. This raises the question: how can trained and verified controll…
Institute for Advanced Studies in Basic Sciences (IASBS)（基础科学高级研究学院）；Charles University（查理大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所）
Nowadays, automobile manufacturers make efforts to develop ways to make cars fully safe. Monitoring driver's actions by computer vision techniques to detect driving mistakes in real-time and then planning for autonomous driving to avoid vehicle collisions is one of the most important issues that has been investigated in the machine vision and Intelligent Transportation Systems (ITS). The main goal of this study is to prevent accidents caused by fatigue, drowsiness, and driver distraction. To avoid these incidents, this paper proposes an integrated safety system that continuously monitors the d…
Tsinghua University（清华大学）；Inner Mongolia University of Technology（内蒙古工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Amphibious ground-aerial vehicles fuse flying and driving modes to enable more flexible air-land mobility and have received growing attention recently. By analyzing the existing amphibious vehicles, we highlight the autonomous fly-driving functionality for the effective uses of amphibious vehicles in complex three-dimensional urban transportation systems. We review and summarize the key enabling technologies for intelligent flying-driving in existing amphibious vehicle designs, identify major technological barriers and propose potential solutions for future research and innovation. This paper …
MIPD: A Multi-sensory Interactive Perception Dataset for Embodied Intelligent Driving
College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）；Institute of Ophthalmology, Beijing Tongren Hospital（北京同仁医院眼科研究所）；College of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；College of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；College of Mechanics and Vehicles, Beijing Institute of Technology（北京理工大学机械与车辆学院）；College of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术学院）
Existing intelligent driving technology often has a problem in balancing smooth driving and fast obstacle avoidance, especially when the vehicle is in a non-structural environment, and is prone to instability in emergency situations. Therefore, this study proposed an autonomous obstacle avoidance control strategy that can effectively guarantee vehicle stability based on Attention-long short-term memory (Attention-LSTM) deep learning model with the idea of humanoid driving. First, we designed the autonomous obstacle avoidance control rules to guarantee the safety of unmanned vehicles. Second, w…
The University of Hong Kong（香港大学）；Inceptio Technology（嬴彻科技）；University of Maryland, College Park（马里兰大学帕克分校）
Recently, there have been many advances in autonomous driving society, attracting a lot of attention from academia and industry. However, existing works mainly focus on cars, extra development is still required for self-driving truck algorithms and models. In this paper, we introduce an intelligent self-driving truck system. Our presented system consists of three main components, 1) a realistic traffic simulation module for generating realistic traffic flow in testing scenarios, 2) a high-fidelity truck model which is designed and evaluated for mimicking real truck response in real-world deplo…
With the increasing complexity of the traffic environment, the importance of safety perception in intelligent driving is growing. Conventional methods in the robust perception of intelligent driving focus on training models with anomalous data, letting the deep neural network decide how to tackle anomalies. However, these models cannot adapt smoothly to the diverse and complex real-world environment. This paper proposes a new type of metric known as Eloss and offers a novel training strategy to empower perception models from the aspect of anomaly detection. Eloss is designed based on an explan…
Honda Research Institute（本田研究所）；School of Informatics, Computing and Engineering, Indiana University（印第安纳大学信息学、计算与工程学院）；Robotics Institute, University of Michigan（密歇根大学机器人研究所）
Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixel-level observations for future vehicle localization. We show that incorporating dense optical flow improves prediction resul…
Beijing Institute of Technology（北京理工大学）；CNRS-UM LIRMM（法国国家科学研究中心-蒙彼利埃大学机器人与微电子实验室）
Developing an intelligent vehicle which can perform human-like actions requires the ability to learn basic driving skills from a large amount of naturalistic driving data. The algorithms will become efficient if we could decompose the complex driving tasks into motion primitives which represent the elementary compositions of driving skills. Therefore, the purpose of this paper is to segment unlabeled trajectory data into a library of motion primitives. By applying a probabilistic inference based on an iterative Expectation-Maximization algorithm, our method segments the collected trajectories …
What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning
Birla Institute of Technology and Science, Pilani（贝拉理工科学学院皮拉尼分校）；University of California, Merced（加州大学默塞德分校）
