Researches show that fatigue driving is one of the important causes of road traffic accidents, so it is of great significance to study the driver fatigue recognition algorithm to improve road traffic safety. In recent years, with the development of deep learning, the field of pattern recognition has made great development. This paper designs a real-time fatigue state recognition algorithm based on spatio-temporal feature sequence, which can be mainly applied to the scene of fatigue driving recognition. The algorithm is divided into three task networks: face detection network, facial landmark detection and head pose estimation network, fatigue recognition network. Experiments show that the algorithm has the advantages of small volume, high speed and high accuracy.
In this paper, we explore different deep learning based approaches to detect driver fatigue. Drowsy driving results in approximately 72,000 crashes and 44,000 injuries every year in the US and detecting drowsiness and alerting the driver can save many lives. There have been many approaches to detect fatigue, of which eye closedness detection is one. We propose a framework to detect eye closedness in a captured camera frame as a gateway for detecting drowsiness. We explore two different datasets to detect eye closedness. We develop an eye model by using new Eye-blink dataset and a face model by…
Indian Institute of Technology (ISM) Dhanbad（印度理工学院（ISM）丹巴德分校）
In recent years, road accidents have increased significantly. One of the major reasons for these accidents, as reported is driver fatigue. Due to continuous and longtime driving, the driver gets exhausted and drowsy which may lead to an accident. Therefore, there is a need for a system to measure the fatigue level of driver and alert him when he/she feels drowsy to avoid accidents. Thus, we propose a system which comprises of a camera installed on the car dashboard. The camera detect the driver's face and observe the alteration in its facial features and uses these features to observe the fati…
In this work various methods and algorithms for face and eyes detection are examined in order to decide which of them are applicable for use in a driver fatigue monitoring system. In the case of face detection the standard Viola-Jones face detector has shown best results, while the method of finding the eye centers by means of gradients has proven to be most appropriate in the case of eyes detection. The later method has also a potential for retrieving behavioral parameters needed for estimation of the level of driver fatigue. This possibility will be examined in future work.
LiteFat: Lightweight Spatio-Temporal Graph Learning for Real-Time Driver Fatigue Detection
School of Computing Technologies, RMIT University（计算技术学院，皇家墨尔本理工大学）；CSIRO’s Data61（CSIRO的数据61）；School of Computer Science, University of Auckland（计算机科学学院，奥克兰大学）；STEM, University of South Australia（STEM，南澳大利亚大学）
Driver fatigue is one of the important factors that cause traffic accidents, and the ever-increasing number due to diminished drivers vigilance level has become a problem of serious concern to society. Drivers with a diminished vigilance level suffer from a marked decline in their abilities of perception, recognition, and vehicle control, and therefore pose serious danger to their own life and the lives of other people. Exhaustion resulting from sleep deprivation or sleep disorders is an important factor in the creasing number of accidents. In this projected work, we discuss the various method…
Tongji University（同济大学）；Key Laboratory of Embedded System and Service Computing, Ministry of Education（嵌入式系统与服务计算教育部重点实验室）
Drowsiness driving is a major cause of traffic accidents and thus numerous previous researches have focused on driver drowsiness detection. Many drive relevant factors have been taken into consideration for fatigue detection and can lead to high precision, but there are still several serious constraints, such as most existing models are environmentally susceptible. In this paper, fatigue detection is considered as temporal action detection problem instead of image classification. The proposed detection system can be divided into four parts: (1) Localize the key patches of the detected driver p…
Intel Deutschland GmbH（英特尔德国有限公司）；Technical University of Munich（慕尼黑工业大学）
Many road accidents occur due to distracted drivers. Today, driver monitoring is essential even for the latest autonomous vehicles to alert distracted drivers in order to take over control of the vehicle in case of emergency. In this paper, a spatio-temporal approach is applied to classify drivers' distraction level and movement decisions using convolutional neural networks (CNNs). We approach this problem as action recognition to benefit from temporal information in addition to spatial information. Our approach relies on features extracted from sparsely selected frames of an action using a pr…
Gwangju Institute of Science and Technology (GIST)（光州科学技术院）；Mokwon University（牧园大学）
We propose a condition-adaptive representation learning framework for the driver drowsiness detection based on 3D-deep convolutional neural network. The proposed framework consists of four models: spatio-temporal representation learning, scene condition understanding, feature fusion, and drowsiness detection. The spatio-temporal representation learning extracts features that can describe motions and appearances in video simultaneously. The scene condition understanding classifies the scene conditions related to various conditions about the drivers and driving situations such as statuses of wea…
YawDD+: Frame-level Annotations for Accurate Yawn Prediction
Embedded Systems Division, Silicon Austria Labs（Silicon Austria Labs嵌入式系统部门）；Institute of Visual Computing, Graz University of Technology（格拉茨技术大学视觉计算研究所）；Department of Computer Science, University of Innsbruck（因斯布鲁克大学计算机科学系）
BIAI Intelligence Biotech Co., Ltd（BIAI智能生物科技有限公司）；BIAI Inc.（BIAI公司）；Shantou University（汕头大学）；Shenzhen Breo Technology CO., LTD.（深圳倍轻松科技股份有限公司）；Harvard University（哈佛大学）；Stealth Mode Startup, INC.（隐身模式初创公司）；Chinese Academy of Sciences（中国科学院）；Massachusetts Institute of Technology（麻省理工学院）
Fatigue detection is valued for people to keep mental health and prevent safety accidents. However, detecting facial fatigue, especially mild fatigue in the real world via machine vision is still a challenging issue due to lack of non-lab dataset and well-defined algorithms. In order to improve the detection capability on facial fatigue that can be used widely in daily life, this paper provided an audiovisual dataset named DLFD (daily-life fatigue dataset) which reflected people's facial fatigue state in the wild. A framework using 3D-ResNet along with non-local attention mechanism was trainin…
Massachusetts Institute of Technology（麻省理工学院）；Toyota Collaborative Safety Research Center（丰田合作安全研究中心）；Touchstone Evaluations
We consider a large dataset of real-world, on-road driving from a 100-car naturalistic study to explore the predictive power of driver glances and, specifically, to answer the following question: what can be predicted about the state of the driver and the state of the driving environment from a 6-second sequence of macro-glances? The context-based nature of such glances allows for application of supervised learning to the problem of vision-based gaze estimation, making it robust, accurate, and reliable in messy, real-world conditions. So, it's valuable to ask whether such macro-glances can be …
Driver inattention assessment has become a very active field in intelligent transportation systems. Based on active sensor Kinect and computer vision tools, we have built an efficient module for detecting driver distraction and recognizing the type of distraction. Based on color and depth map data from the Kinect, our system is composed of four sub-modules. We call them eye behavior (detecting gaze and blinking), arm position (is the right arm up, down, right of forward), head orientation, and facial expressions. Each module produces relevant information for assessing driver inattention. They …
Xperi Corporation（Xperi公司）；National University of Ireland Galway（爱尔兰国立高威大学）；Prophesee（Prophesee公司）
Event cameras contain emerging, neuromorphic vision sensors that capture local light intensity changes at each pixel, generating a stream of asynchronous events. This way of acquiring visual information constitutes a departure from traditional frame based cameras and offers several significant advantages: low energy consumption, high temporal resolution, high dynamic range and low latency. Driver monitoring systems (DMS) are in-cabin safety systems designed to sense and understand a drivers physical and cognitive state. Event cameras are particularly suited to DMS due to their inherent advanta…
Mercedes-Benz AG（梅赛德斯-奔驰股份公司）；University of Tübingen（图宾根大学）
Modern advanced driver-assistance systems analyze the driving performance to gather information about the driver's state. Such systems are able, for example, to detect signs of drowsiness by evaluating the steering or lane keeping behavior and to alert the driver when the drowsiness state reaches a critical level. However, these kinds of systems have no access to direct cues about the driver's state. Hence, the aim of this work is to extend the driver drowsiness detection in vehicles using signals of a driver monitoring camera. For this purpose, 35 features related to the driver's eye blinking…
The alertness level of drivers can be estimated with the use of computer vision based methods. The level of fatigue can be found from the value of PERCLOS. It is the ratio of closed eye frames to the total frames processed. The main objective of the thesis is the design and implementation of real-time algorithms for measurement of PERCLOS. In this work we have developed a real-time system which is able to process the video onboard and to alarm the driver in case the driver is in alert. For accurate estimation of PERCLOS the frame rate should be greater than 4 and accuracy should be greater tha…
Vignan Institute of Technology and Sciences（维格纳科技学院）；VITS（维格纳科技学院）
Driving support systems, such as car navigation systems are becoming common and they support driver in several aspects. Non-intrusive method of detecting Fatigue and drowsiness based on eye-blink count and eye directed instruction controlhelps the driver to prevent from collision caused by drowsy driving. Eye detection and tracking under various conditions such as illumination, background, face alignment and facial expression makes the problem complex.Neural Network based algorithm is proposed in this paper to detect the eyes efficiently. In the proposed algorithm, first the neural Network is …
Graph-Based Recognition of Simulated Train-Driver States From Facial and Upper-Body Keypoints
Istituto Italiano di Tecnologia（意大利技术研究院）；Imperial College London（帝国理工学院）；Italian National Institute for Insurance against Accidents at Work (INAIL)（意大利国家工伤事故保险研究所（INAIL））
Institute for Advanced Studies in Basic Sciences (IASBS)（基础科学高级研究学院）；Charles University（查理大学）；Institute for Research in Fundamental Sciences (IPM)（基础科学研究所）
Nowadays, automobile manufacturers make efforts to develop ways to make cars fully safe. Monitoring driver's actions by computer vision techniques to detect driving mistakes in real-time and then planning for autonomous driving to avoid vehicle collisions is one of the most important issues that has been investigated in the machine vision and Intelligent Transportation Systems (ITS). The main goal of this study is to prevent accidents caused by fatigue, drowsiness, and driver distraction. To avoid these incidents, this paper proposes an integrated safety system that continuously monitors the d…
Awareness detection technologies have been gaining traction in a variety of enterprises; most often used for driver fatigue detection, recent research has shifted towards using computer vision technologies to analyze user attention in environments such as online classrooms. This paper aims to extend previous research on distraction detection by analyzing which visual features contribute most to predicting awareness and fatigue. We utilized the open source facial analysis toolkit OpenFace in order to analyze visual data of subjects at varying levels of attentiveness. Then, using a Support-Vecto…
Chalmers University of Technology（查尔姆斯理工大学）；Volvo Cars Technology USA（沃尔沃汽车美国技术公司）；Volvo Car Corporation（沃尔沃汽车公司）；University of California, Berkeley（加州大学伯克利分校）
Pedestrian intention recognition is very important to develop robust and safe autonomous driving (AD) and advanced driver assistance systems (ADAS) functionalities for urban driving. In this work, we develop an end-to-end pedestrian intention framework that performs well on day- and night- time scenarios. Our framework relies on objection detection bounding boxes combined with skeletal features of human pose. We study early, late, and combined (early and late) fusion mechanisms to exploit the skeletal features and reduce false positives as well to improve the intention prediction performance. …
Nagoya University（名古屋大学）；UT Dallas（得克萨斯大学达拉斯分校）；Daido University（大同大学）；Tier IV Inc.（Tier IV公司）
Advanced driver assistance and automated driving systems rely on risk estimation modules to predict and avoid dangerous situations. Current methods use expensive sensor setups and complex processing pipeline, limiting their availability and robustness. To address these issues, we introduce a novel deep learning based action recognition framework for classifying dangerous lane change behavior in short video clips captured by a monocular camera. We designed a deep spatiotemporal classification network that uses pre-trained state-of-the-art instance segmentation network Mask R-CNN as its spatial …
There is significant progress in recognizing traditional human activities from videos focusing on highly distinctive actions involving discriminative body movements, body-object and/or human-human interactions. Driver's activities are different since they are executed by the same subject with similar body parts movements, resulting in subtle changes. To address this, we propose a novel framework by exploiting the spatiotemporal attention to model the subtle changes. Our model is named Coarse Temporal Attention Network (CTA-Net), in which coarse temporal branches are introduced in a trainable g…
Driver attention prediction is becoming an essential research problem in human-like driving systems. This work makes an attempt to predict the driver attention in driving accident scenarios (DADA). However, challenges tread on the heels of that because of the dynamic traffic scene, intricate and imbalanced accident categories. In this work, we design a semantic context induced attentive fusion network (SCAFNet). We first segment the RGB video frames into the images with different semantic regions (i.e., semantic images), where each region denotes one kind of semantic categories of the scene (e…
A Contextual Analysis of Driver-Facing and Dual-View Video Inputs for Distraction Detection in Naturalistic Driving Environments
Environmental Engineering The University of Memphis Memphis, TN, USA Email（环境工程 明尼苏达大学 内华达州法戈 邮箱）；Environmental Engineering North Dakota State University Fargo, ND, USA Email（环境工程 内华达州立大学 内华达州法戈 邮箱）
Lane detection is one of the indispensable and key elements of self-driving environmental perception. Many lane detection models have been proposed, solving lane detection under challenging conditions, including intersection merging and splitting, curves, boundaries, occlusions and combinations of scene types. Nevertheless, lane detection will remain an open problem for some time to come. The ability to cope well with those challenging scenes impacts greatly the applications of lane detection on advanced driver assistance systems (ADASs). In this paper, a spatio-temporal network with double Co…
Driver drowsiness increases crash risk, leading to substantial road trauma each year. Drowsiness detection methods have received considerable attention, but few studies have investigated the implementation of a detection approach on a mobile phone. Phone applications reduce the need for specialised hardware and hence, enable a cost-effective roll-out of the technology across the driving population. While it has been shown that three-dimensional (3D) operations are more suitable for spatiotemporal feature learning, current methods for drowsiness detection commonly use frame-based, multi-step ap…
Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision
State Key Laboratory of Advanced Design and Manufacturing for Vehicle Body（汽车车身先进设计与制造国家重点实验室）；College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与运载工程学院）；Faculty of Engineering and IT, University of Technology Sydney（悉尼科技大学工程与信息技术学院）；College of Automotive and Mechanical Engineering, ChangSha University of Science Technology（长沙理工大学汽车与机械工程学院）
Cornell University（康奈尔大学）；Stanford University（斯坦福大学）；Brain Of Things Inc.（Brain Of Things公司）
Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns…
