This paper is reprinted from, "VRUNet: Multi-Task Learning Model for Intent Prediction of Vulnerable Road Users, IS&T Electronic Imaging: Autonomous Vehicles and Machines 2020 Proceedings, (IS&T, Springfield, VA, 2020) page 109-1-10. DOI: 10.2352/ISSN.2470-1173.2020.16.AVM-109." Reprinted with permission of The Society for Imaging Science and Technology, holders of the 2020 copyright
Advanced perception and path planning are at the core for any self-driving vehicle. Autonomous vehicles need to understand the scene and intentions of other road users for safe motion planning. For urban use cases it is very important to perceive and predict the intentions of pedestrians, cyclists, scooters, etc., classified as vulnerable road users (VRU). Intent is a combination of pedestrian activities and long term trajectories defining their future motion. In this paper we propose a multi-task learning model to predict pedestrian actions, crossing intent and forecast their future path from video sequences. We have trained the model on naturalistic driving open-source JAAD dataset, which is rich in behavioral annotations and real world scenarios. Experimental results show state-of-the-art performance on JAAD dataset and how we can benefit from jointly learning and predicting actions and trajectories using 2D human pose features and scene context.
Following detection and tracking of traffic actors, prediction of their future motion is the next critical component of a self-driving vehicle (SDV) technology, allowing the SDV to operate safely and efficiently in its environment. This is particularly important when it comes to vulnerable road users (VRUs), such as pedestrians and bicyclists. These actors need to be handled with special care due to an increased risk of injury, as well as the fact that their behavior is less predictable than that of motorized actors. To address this issue, in the current study we present a deep learning-based …
Leibniz University Hannover（莱布尼茨汉诺威大学）；Nagoya University（名古屋大学）；University of Human Environments（人类环境大学）
Intersections where vehicles are permitted to turn and interact with vulnerable road users (VRUs) like pedestrians and cyclists are among some of the most challenging locations for automated and accurate recognition of road users' behavior. In this paper, we propose a deep conditional generative model for interaction detection at such locations. It aims to automatically analyze massive video data about the continuity of road users' behavior. This task is essential for many intelligent transportation systems such as traffic safety control and self-driving cars that depend on the understanding o…
Predictions of driver's intentions and their behaviors using the road is of great importance for planning and decision making processes of autonomous driving vehicles. In particular, relatively short-term driving intentions are the fundamental units that constitute more sophisticated driving goals, behaviors, such as overtaking the slow vehicle in front, exit or merge onto a high way, etc. While it is not uncommon that most of the time human driver can rationalize, in advance, various on-road behaviors, intentions, as well as the associated risks, aggressiveness, reciprocity characteristics, e…
University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；Honda Research Institute, USA（本田美国研究院）
Road-users are a critical part of decision-making for both self-driving cars and driver assistance systems. Some road-users, however, are more important for decision-making than others because of their respective intentions, ego vehicle's intention and their effects on each other. In this paper, we propose a novel architecture for road-user importance estimation which takes advantage of the local and global context of the scene. For local context, the model exploits the appearance of the road users (which captures orientation, intention, etc.) and their location relative to ego-vehicle. The gl…
OnSiteVRU: A High-Resolution Trajectory Dataset for High-Density Vulnerable Road Users
School of Traffic；Transportation Engineering, Changsha University of Science \& Technology, No. 960, 2nd section of Wanjiali Road, Changsha, Hunan Province, 410114, China；Key Laboratory of Road；Traffic Engineering of Ministry of Education \& School of Transportation Engineering, Tongji University. No.4800 Cao’an Road, Shanghai, 201804, China.；School of Automobile；Transportation, Xihua University Chengdu Sichuan, China
Johannes Kepler University Linz（约翰开普勒林茨大学）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Ural Federal University（乌拉尔联邦大学）
Rapid advancements in driver-assistance technology will lead to the integration of fully autonomous vehicles on our roads that will interact with other road users. To address the problem that driverless vehicles make interaction through eye contact impossible, we describe a framework for estimating the crossing intentions of pedestrians in order to reduce the uncertainty that the lack of eye contact between road users creates. The framework was deployed in a real vehicle and tested with three experimental cases that showed a variety of communication messages to pedestrians in a shared space sc…
Johns Hopkins University（约翰斯·霍普金斯大学）；National Yang Ming Chiao Tung University（阳明交通大学）
In this work, we tackle two vital tasks in automated driving systems, i.e., driver intent prediction and risk object identification from egocentric images. Mainly, we investigate the question: what would be good road scene-level representations for these two tasks? We contend that a scene-level representation must capture higher-level semantic and geometric representations of traffic scenes around ego-vehicle while performing actions to their destinations. To this end, we introduce the representation of semantic regions, which are areas where ego-vehicles visit while taking an afforded action …
Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Anticipating the intentions of vulnerable road users (VRUs) such as pedestrians and cyclists is critical for performing safe and comfortable driving maneuvers. This is the case for human driving and, thus, should be taken into account by systems providing any level of driving assistance, from advanced driver assistant systems (ADAS) to fully autonomous vehicles (AVs). In this paper, we show how the latest advances on monocular vision-based human pose estimation, i.e. those relying on deep Convolutional Neural Networks (CNNs), enable to recognize the intentions of such VRUs. In the case of cycl…
Tshwane University of Technology（茨瓦内理工大学）；Central University of Technology（中央理工大学）
Autonomous driving systems are broadly used equipment in the industries and in our daily lives, they assist in production, but are majorly used for exploration in dangerous or unfamiliar locations. Thus, for a successful exploration, navigation plays a significant role. Road detection is an essential factor that assists autonomous robots achieved perfect navigation. Various techniques using camera sensors have been proposed by numerous scholars with inspiring results, but their techniques are still vulnerable to these environmental noises: rain, snow, light intensity and shadow. In addressing …
Socially-Aware Autonomous Driving: Inferring Yielding Intentions for Safer Interactions
School of Mechanical and Aerospace Engineering, Queen’s University Belfast（女王大学贝尔法斯特机械与航空航天工程学院）；Department of Mechanical and Industrial Engineering, Concordia University（Concordia大学机械与工业工程系）；State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, College of Mechanical and Vehicle Engineering, Hunan University（湖南大学车辆先进设计与制造技术国家重点实验室，机械与车辆工程学院）；James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）
This paper describes a novel method for allowing an autonomous ground vehicle to predict the intent of other agents in an urban environment. This method, termed the cognitive driving framework, models both the intent and the potentially false beliefs of an obstacle vehicle. By modeling the relationships between these variables as a dynamic Bayesian network, filtering can be performed to calculate the intent of the obstacle vehicle as well as its belief about the environment. This joint knowledge can be exploited to plan safer and more efficient trajectories when navigating in an urban environm…
VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis
German University in Cairo（埃及德国大学）；Faculty of Media Engineering and Technology（媒体工程与技术学院）；Computer Science and Engineering Department（计算机科学与工程系）；C-DRiVeS Lab（C-DRiVeS实验室）；Cognitive Driving Research in Vehicular Systems（车载系统认知驾驶研究）
ControlMTR: Control-Guided Motion Transformer with Scene-Compliant Intention Points for Feasible Motion Prediction
Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）；College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Moovita Pte Ltd（Moovita公司）
Accurately forecasting the future movements of surrounding vehicles is essential for safe and efficient operations of autonomous driving cars. This task is difficult because a vehicle's moving trajectory is greatly determined by its driver's intention, which is often hard to estimate. By leveraging attention mechanisms along with long short-term memory (LSTM) networks, this work learns the relation between a driver's intention and the vehicle's changing positions relative to road infrastructures, and uses it to guide the prediction. Different from other state-of-the-art solutions, our work tre…
One challenging aspect of the Connected and Automated Vehicle (CAV) operation in mixed traffic is the development of a situation-awareness module for CAVs. While operating on public roads, CAVs need to assess their surroundings, especially the intentions of non-CAVs. Generally, CAVs demonstrate a defensive driving behavior, and CAVs expect other non-autonomous entities on the road will follow the traffic rules or common driving behavior. However, the presence of aggressive human drivers in the surrounding environment, who may not follow traffic rules and behave abruptly, can lead to serious sa…
Understanding complex scenarios from in-vehicle cameras is essential for safely operating autonomous driving systems in densely populated areas. Among these, intersection areas are one of the most critical as they concentrate a considerable number of traffic accidents and fatalities. Detecting and understanding the scene configuration of these usually crowded areas is then of extreme importance for both autonomous vehicles and modern ADAS aimed at preventing road crashes and increasing the safety of vulnerable road users. This work investigates inter-section classification from RGB images usin…
Department of Computer Science and Applied Physics, Galway-Mayo Institute of Technology（戈尔韦-梅奥理工学院计算机科学与应用物理系）；Galway-Mayo Institute of Technology（戈尔韦-梅奥理工学院）
Correctly identifying vulnerable road users (VRUs), e.g. cyclists and pedestrians, remains one of the most challenging environment perception tasks for autonomous vehicles (AVs). This work surveys the current state-of-the-art in VRU detection, covering topics such as benchmarks and datasets, object detection techniques and relevant machine learning algorithms. The article concludes with a discussion of remaining open challenges and promising future research directions for this domain.
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
In order to protect vulnerable road users (VRUs), such as pedestrians or cyclists, it is essential that intelligent transportation systems (ITS) accurately identify them. Therefore, datasets used to train perception models of ITS must contain a significant number of vulnerable road users. However, data protection regulations require that individuals are anonymized in such datasets. In this work, we introduce a novel deep learning-based pipeline for face anonymization in the context of ITS. In contrast to related methods, we do not use generative adversarial networks (GANs) but build upon recen…
Department of Electrical Engineering and Computer Science, York University（约克大学电气工程与计算机科学系）；York University（约克大学）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Toronto（多伦多）；ON, Canada（加拿大安大略省）
One of the major challenges that autonomous cars are facing today is driving in urban environments. To make it a reality, autonomous vehicles require the ability to communicate with other road users and understand their intentions. Such interactions are essential between the vehicles and pedestrians as the most vulnerable road users. Understanding pedestrian behavior, however, is not intuitive and depends on various factors such as demographics of the pedestrians, traffic dynamics, environmental conditions, etc. In this paper, we identify these factors by surveying pedestrian behavior studies,…
California PATH, University of California, Berkeley（加州大学伯克利分校加州合作伙伴先进运输技术项目（PATH））；Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）
As autonomous vehicles (AVs) need to interact with other road users, it is of importance to comprehensively understand the dynamic traffic environment, especially the future possible trajectories of surrounding vehicles. This paper presents an algorithm for long-horizon trajectory prediction of surrounding vehicles using a dual long short term memory (LSTM) network, which is capable of effectively improving prediction accuracy in strongly interactive driving environments. In contrast to traditional approaches which require trajectory matching and manual feature selection, this method can autom…
Mind2Drive: Predicting Driver Intentions from EEG in Real-world On-Road Driving
Department of Computer Science, Imam Mohammad Ibn Saud Islamic University（伊玛姆·莫哈默德·伊本·沙特伊斯兰大学计算机科学系）；Department of Computer Science, Princess Nourah bint Abdulrahman University（努拉·本·阿卜杜勒拉赫曼大学计算机科学系）
Seoul National University（首尔大学）；Dongguk University（东国大学）
Autonomous vehicles (AVs) can prevent collisions by understanding pedestrian intention. We conducted a virtual reality experiment with 39 participants and measured crossing times (seconds) and head orientation (yaw degrees). We manipulated AV yielding behavior (no-yield, slow-yield, and fast-yield) and the AV size (small, medium, and large). Using machine learning approach, we classified head orientation change of pedestrians by time into 6 clusters of patterns. Results indicate that pedestrian head orientation change was influenced by AV yielding behavior as well as the size of the AV. Partic…
Vehicle-in-Virtual-Environment (VVE) Method for Developing and Evaluating VRU Safety of Connected and Autonomous Driving with Focus on Bicyclist Safety
A significant barrier to deploying autonomous vehicles (AVs) on a massive scale is safety assurance. Several technical challenges arise due to the uncertain environment in which AVs operate such as road and weather conditions, errors in perception and sensory data, and also model inaccuracy. In this paper, we propose a system architecture for risk-aware AVs capable of reasoning about uncertainty and deliberately bounding the risk of collision below a given threshold. We discuss key challenges in the area, highlight recent research developments, and propose future research directions in three s…
This article describes an approach to detect the wearing location of smart devices worn by pedestrians and cyclists. The detection, which is based solely on the sensors of the smart devices, is important context-information which can be used to parametrize subsequent algorithms, e.g. for dead reckoning or intention detection to improve the safety of vulnerable road users. The wearing location recognition can in terms of Organic Computing (OC) be seen as a step towards self-awareness and self-adaptation. For the wearing location detection a two-stage process is presented. It is subdivided into …
University of Applied Sciences Aschaffenburg（阿沙芬堡应用科学大学）；University of Kassel（卡塞尔大学）
In this article, an approach for probabilistic trajectory forecasting of vulnerable road users (VRUs) is presented, which considers past movements and the surrounding scene. Past movements are represented by 3D poses reflecting the posture and movements of individual body parts. The surrounding scene is modeled in the form of semantic maps showing, e.g., the course of streets, sidewalks, and the occurrence of obstacles. The forecasts are generated in grids discretizing the space and in the form of arbitrary discrete probability distributions. The distributions are evaluated in terms of their r…
University of Applied Sciences Aschaffenburg（阿沙芬堡应用科学大学）；University of Kassel（卡塞尔大学）；Ulm University（乌尔姆大学）
Avoiding collisions with vulnerable road users (VRUs) using sensor-based early recognition of critical situations is one of the manifold opportunities provided by the current development in the field of intelligent vehicles. As especially pedestrians and cyclists are very agile and have a variety of movement options, modeling their behavior in traffic scenes is a challenging task. In this article we propose movement models based on machine learning methods, in particular artificial neural networks, in order to classify the current motion state and to predict the future trajectory of VRUs. Both…
Center for Research in Mathematics CIMAT AC（数学研究中心 CIMAT AC）；Cátedras CONACYT（CONACYT 教席）
In this work, orientation detection using Deep Learning is acknowledged for a particularly vulnerable class of road users,the cyclists. Knowing the cyclists' orientation is of great relevance since it provides a good notion about their future trajectory, which is crucial to avoid accidents in the context of intelligent transportation systems. Using Transfer Learning with pre-trained models and TensorFlow, we present a performance comparison between the main algorithms reported in the literature for object detection,such as SSD, Faster R-CNN and R-FCN along with MobilenetV2, InceptionV2, ResNet…
TrajFusionNet: Pedestrian Crossing Intention Prediction via Fusion of Sequential and Visual Trajectory Representations
Perception, Robotics and Intelligent Machines Research Group (PRIME) Department of Computer Science, Université de Moncton, Canada（感知、机器人与智能机器研究组（PRIME）计算机科学系，新不伦瑞克大学）
Honda Research Institute, USA（美国本田研究院）；Sungkyunkwan University（成均馆大学）
We propose a Deep RObust Goal-Oriented trajectory prediction Network (DROGON) for accurate vehicle trajectory prediction by considering behavioral intentions of vehicles in traffic scenes. Our main insight is that the behavior (i.e., motion) of drivers can be reasoned from their high level possible goals (i.e., intention) on the road. To succeed in such behavior reasoning, we build a conditional prediction model to forecast goal-oriented trajectories with the following stages: (i) relational inference where we encode relational interactions of vehicles using the perceptual context; (ii) intent…
