Naturalistic driving data (NDD) is an important source of information to understand crash causation and human factors and to further develop crash avoidance countermeasures. Videos recorded while driving are often included in such datasets. While there is often a large amount of video data in NDD, only a small portion of them can be annotated by human coders and used for research, which underuses all video data. In this paper, we explored a computer vision method to automatically extract the information we need from videos. More specifically, we developed a 3D ConvNet algorithm to automatically extract cell-phone-related behaviors from videos. The experiments show that our method can extract chunks from videos, most of which (~79%) contain the automatically labeled cell phone behaviors. In conjunction with human review of the extracted chunks, this approach can find cell-phone-related driver behaviors much more efficiently than simply viewing video.
Boston University（波士顿大学）；Georgetown University（乔治城大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Modern deep neural networks tend to be evaluated on static test sets. One shortcoming of this is the fact that these deep neural networks cannot be easily evaluated for robustness issues with respect to specific scene variations. For example, it is hard to study the robustness of these networks to variations of object scale, object pose, scene lighting and 3D occlusions. The main reason is that collecting real datasets with fine-grained naturalistic variations of sufficient scale can be extremely time-consuming and expensive. In this work, we present Counterfactual Simulation Testing, a counte…
Understanding the intentions of drivers at intersections is a critical component for autonomous vehicles. Urban intersections that do not have traffic signals are a common epicentre of highly variable vehicle movement and interactions. We present a method for predicting driver intent at urban intersections through multi-modal trajectory prediction with uncertainty. Our method is based on recurrent neural networks combined with a mixture density network output layer. To consolidate the multi-modal nature of the output probability distribution, we introduce a clustering algorithm that extracts t…
Recently, multiple naturalistic traffic datasets of human-driven trajectories have been published (e.g., highD, NGSim, and pNEUMA). These datasets have been used in studies that investigate variability in human driving behavior, for example for scenario-based validation of autonomous vehicle (AV) behavior, modeling driver behavior, or validating driver models. Thus far, these studies focused on the variability on an operational level (e.g., velocity profiles during a lane change), not on a tactical level (i.e., to change lanes or not). Investigating the variability on both levels is necessary …
Continuous estimation the driver's take-over readiness is critical for safe and timely transfer of control during the failure modes of autonomous vehicles. In this paper, we propose a data-driven approach for estimating the driver's take-over readiness based purely on observable cues from in-vehicle vision sensors. We present an extensive naturalistic drive dataset of drivers in a conditionally autonomous vehicle running on Californian freeways. We collect subjective ratings for the driver's take-over readiness from multiple human observers viewing the sensor feed. Analysis of the ratings in t…
PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Sichuan Agricultural University（四川农业大学）
The most common type of accident on the road is a rear-end crash. These crashes have a significant negative impact on traffic flow and are frequently fatal. To gain a more practical understanding of these scenarios, it is necessary to accurately model car following behaviors that result in rear-end crashes. Numerous studies have been carried out to model drivers' car-following behaviors; however, the majority of these studies have relied on simulated data, which may not accurately represent real-world incidents. Furthermore, most studies are restricted to modeling the ego vehicle's acceleratio…
A Contextual Analysis of Driver-Facing and Dual-View Video Inputs for Distraction Detection in Naturalistic Driving Environments
Environmental Engineering The University of Memphis Memphis, TN, USA Email（环境工程 明尼苏达大学 内华达州法戈 邮箱）；Environmental Engineering North Dakota State University Fargo, ND, USA Email（环境工程 内华达州立大学 内华达州法戈 邮箱）
In this paper, we presented a preliminary study for tactical driver behavior detection from untrimmed naturalistic driving recordings. While supervised learning based detection is a common approach, it suffers when labeled data is scarce. Manual annotation is both time-consuming and expensive. To emphasize this problem, we experimented on a 104-hour real-world naturalistic driving dataset with a set of predefined driving behaviors annotated. There are three challenges in the dataset. First, predefined driving behaviors are sparse in a naturalistic driving setting. Second, the distribution of d…
Fusing Driver Perceived and Physical Risk for Safety Critical Scenario Screening in Autonomous Driving
Guangdong Provincial Key Laboratory of Intelligent Transportation System, School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（广东省智能交通系统重点实验室，智能系统工程学院，中山大学深圳校区）；school of Transportation and Logistics, Southwest Jiaotong University（交通运输与物流学院，西南交通大学）
Driver Age and Its Effect on Key Driving Metrics: Insights from Dynamic Vehicle Data
Iowa State University of Science and Technology（爱荷华州立科技大学）；National Institute on Aging（国家老龄化研究所）；Toyota Motor Engineering & Manufacturing North America Inc.（丰田北美汽车工程制造公司）
LISA: Laboratory for Intelligent & Safe Automobiles（智能与安全汽车实验室）；University of California San Diego（加州大学圣地亚哥分校）
In the context of autonomous driving, where humans may need to take over in the event where the computer may issue a takeover request, a key step towards driving safety is the monitoring of the hands to ensure the driver is ready for such a request. This work, focuses on the first step of this process, which is to locate the hands. Such a system must work in real-time and under varying harsh lighting conditions. This paper introduces a fast ConvNet approach, based on the work of original work of OpenPose for full body joint estimation. The network is modified with fewer parameters and retraine…
Massachusetts Institute of Technology (MIT)（麻省理工学院（MIT））；Touchstone Evaluations, Inc.（托奇斯通评估公司）
For the foreseeble future, human beings will likely remain an integral part of the driving task, monitoring the AI system as it performs anywhere from just over 0% to just under 100% of the driving. The governing objectives of the MIT Autonomous Vehicle Technology (MIT-AVT) study are to (1) undertake large-scale real-world driving data collection that includes high-definition video to fuel the development of deep learning based internal and external perception systems, (2) gain a holistic understanding of how human beings interact with vehicle automation technology by integrating video data wi…
Driving as a Diagnostic Tool: Scenario-based Cognitive Assessment in Older Drivers from Driving Video
organization= Department of Electrical \& Computer Engineering , addressline= Iowa State University , city= Ames , state= IA , postcode= 50011 , country= USA；organization= Institute for Transportation , addressline= Iowa State University , city= Ames , state= IA , postcode= 50010 , country= USA；Brain Health Lab , addressline= University of Nebraska Medical Center , city= Omaha , state= NE , postcode= 68198 , country= USA；organization= Department of Mechanical Engineering , addressline= Iowa State University , city= Ames , state= IA , postcode= 50011 , country= USA
Driving in a human-like manner is important for an autonomous vehicle to be a smart and predictable traffic participant. To achieve this goal, parameters of the motion planning module should be carefully tuned, which needs great effort and expert knowledge. In this study, a method of learning cost parameters of a motion planner from naturalistic driving data is proposed. The learning is achieved by encouraging the selected trajectory to approximate the human driving trajectory under the same traffic situation. The employed motion planner follows a widely accepted methodology that first samples…
BMW of North America（北美宝马公司）；Graz University of Technology（格拉茨工业大学）
Collecting realistic driving trajectories is crucial for training machine learning models that imitate human driving behavior. Most of today's autonomous driving datasets contain only a few trajectories per location and are recorded with test vehicles that are cautiously driven by trained drivers. In particular in interactive scenarios such as highway merges, the test driver's behavior significantly influences other vehicles. This influence prevents recording the whole traffic space of human driving behavior. In this work, we present a novel methodology to extract trajectories of traffic objec…
Beijing Institute of Technology（北京理工大学）；University of California, Berkeley（加州大学伯克利分校）；University of Michigan（密歇根大学）
Analysis and recognition of driving styles are profoundly important to intelligent transportation and vehicle calibration. This paper presents a novel driving style analysis framework using the primitive driving patterns learned from naturalistic driving data. In order to achieve this, first, a Bayesian nonparametric learning method based on a hidden semi-Markov model (HSMM) is introduced to extract primitive driving patterns from time series driving data without prior knowledge of the number of these patterns. In the Bayesian nonparametric approach, we utilize a hierarchical Dirichlet process…
School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
The urban intersection is a typically dynamic and complex scenario for intelligent vehicles, which exists a variety of driving behaviors and traffic participants. Accurately modelling the driver behavior at the intersection is essential for intelligent transportation systems (ITS). Previous researches mainly focus on using attention mechanism to model the degree of correlation. In this research, a canonical correlation analysis (CCA)-based framework is proposed. The value of canonical correlation is used for feature selection. Gaussian mixture model and Gaussian process regression are applied …
University of Missouri-Columbia（密苏里大学哥伦比亚分校）；Syracuse University-USA（美国雪城大学）；University of Nebraska Medical Center（内布拉斯加大学医学中心）；Iowa State University（爱荷华州立大学）
The current paper implements a methodology for automatically detecting vehicle maneuvers from vehicle telemetry data under naturalistic driving settings. Previous approaches have treated vehicle maneuver detection as a classification problem, although both time series segmentation and classification are required since input telemetry data is continuous. Our objective is to develop an end-to-end pipeline for frame-by-frame annotation of naturalistic driving studies videos into various driving events including stop and lane keeping events, lane changes, left-right turning movements, and horizont…
University of Michigan（密歇根大学）；University of California at Berkeley（加州大学伯克利分校）
Developing an automated vehicle, that can handle complicated driving scenarios and appropriately interact with other road users, requires the ability to semantically learn and understand driving environment, oftentimes, based on analyzing massive amounts of naturalistic driving data. An important paradigm that allows automated vehicles to both learn from human drivers and gain insights is understanding the principal compositions of the entire traffic, termed as traffic primitives. However, the exploding data growth presents a great challenge in extracting primitives from high-dimensional time-…
Spatial-Temporal Perception with Causal Inference for Naturalistic Driving Action Recognition
School of Mechanical Engineering, Nanjing University of Science and Technology（南京理工大学机械工程学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学先进技术学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Driver gaze has been shown to be an excellent surrogate for driver attention in intelligent vehicles. With the recent surge of highly autonomous vehicles, driver gaze can be useful for determining the handoff time to a human driver. While there has been significant improvement in personalized driver gaze zone estimation systems, a generalized system which is invariant to different subjects, perspectives and scales is still lacking. We take a step towards this generalized system using Convolutional Neural Networks (CNNs). We finetune 4 popular CNN architectures for this task, and provide extens…
Massachusetts Institute of Technology（麻省理工学院）；Toyota Collaborative Safety Research Center（丰田合作安全研究中心）；Touchstone Evaluations
We consider a large dataset of real-world, on-road driving from a 100-car naturalistic study to explore the predictive power of driver glances and, specifically, to answer the following question: what can be predicted about the state of the driver and the state of the driving environment from a 6-second sequence of macro-glances? The context-based nature of such glances allows for application of supervised learning to the problem of vision-based gaze estimation, making it robust, accurate, and reliable in messy, real-world conditions. So, it's valuable to ask whether such macro-glances can be …
Transportation & Autonomous Systems Institute (TASI), IUPUI（印第安纳大学与普渡大学印第安纳波利斯联合分校交通与自动驾驶系统研究所）；Collaborative Safety Research Center (CSRC), Toyota Motor North America (TMNA)（丰田汽车北美公司合作安全研究中心）
As one of the most popular micro-mobility options, e-scooters are spreading in hundreds of big cities and college towns in the US and worldwide. In the meantime, e-scooters are also posing new challenges to traffic safety. In general, e-scooters are suggested to be ridden in bike lanes/sidewalks or share the road with cars at the maximum speed of about 15-20 mph, which is more flexible and much faster than the pedestrains and bicyclists. These features make e-scooters challenging for human drivers, pedestrians, vehicle active safety modules, and self-driving modules to see and interact. To stu…
AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception
Fudan University（复旦大学）；Institute of Meta-Medical（元医学研究所）；Engineering Research Center of AI and Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）；AI and Unmanned Systems Engineering Research Center of Jilin Province（吉林省人工智能与无人系统工程研究中心）；Boli Technology Co., Ltd.（博力科技有限公司）
Volpe National Transportation Systems Center（沃尔普国家运输系统中心）；U.S. Department of Transportation（美国交通部）
Naturalistic driving studies seek to perform the observations of human driver behavior in the variety of environmental conditions necessary to analyze, understand and predict that behavior using statistical and physical models. The second Strategic Highway Research Program (SHRP 2) funds a number of transportation safety-related projects including its primary effort, the Naturalistic Driving Study (NDS), and an effort supplementary to the NDS, the Roadway Information Database (RID). This work seeks to expand the range of answerable research questions that researchers might pose to the NDS and …
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Honda Research Institute（本田研究所）
Anomaly driving detection is an important problem in advanced driver assistance systems (ADAS). It is important to identify potential hazard scenarios as early as possible to avoid potential accidents. This study proposes an unsupervised method to quantify driving anomalies using a conditional generative adversarial network (GAN). The approach predicts upcoming driving scenarios by conditioning the models on the previously observed signals. The system uses the difference of the output from the discriminator between the predicted and actual signals as a metric to quantify the anomaly degree of …
