驾驶员动作识别旨在准确识别驾驶员的行为，对于提升人车交互和保障驾驶安全至关重要。与通用动作识别不同，驾驶员所处环境通常具有挑战性，往往昏暗无光；且随着传感器技术的发展，红外（IR）相机、深度相机等各类相机已被用于分析驾驶员行为。因此，本文提出一种新型多模态融合Transformer，名为MultiFuser，它能识别多模态车内视频之间的跨模态关联与交互，并自适应整合不同模态以获得更优表征。具体而言，MultiFuser由多层双分解模块（Bi-decomposed Modules）构成以建模时空特征，还配备一个模态合成器用于多模态特征整合。每个双分解模块包含一个用于提取模态特定特征的模态专长ViT块（Modal Expertise ViT block），以及一个用于高效跨模态融合的逐块自适应融合块（Patch-wise Adaptive Fusion block）。我们在Drive&Act数据集上开展了大量实验，结果证明了所提方法的有效性。
英文摘要：
Driver action recognition, aiming to accurately identify drivers' behaviours, is crucial for enhancing driver-vehicle interactions and ensuring driving safety. Unlike general action recognition, drivers' environments are often challenging, being gloomy and dark, and with the development of sensors, various cameras such as IR and depth cameras have emerged for analyzing drivers' behaviors. Therefore, in this paper, we propose a novel multimodal fusion transformer, named MultiFuser, which identifies cross-modal interrelations and interactions among multimodal car cabin videos and adaptively integrates different modalities for improved representations. Specifically, MultiFuser comprises layers of Bi-decomposed Modules to model spatiotemporal features, with a modality synthesizer for multimodal features integration. Each Bi-decomposed Module includes a Modal Expertise ViT block for extracting modality-specific features and a Patch-wise Adaptive Fusion block for efficient cross-modal fusion. Extensive experiments are conducted on Drive&Act dataset and the results demonstrate the efficacy of our proposed approach.
DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）；Singapore University of Technology and Design (SUTD), Singapore（新加坡科技设计大学）；Continental Automotive Singapore Pte. Ltd.
A Contextual Analysis of Driver-Facing and Dual-View Video Inputs for Distraction Detection in Naturalistic Driving Environments
Environmental Engineering The University of Memphis Memphis, TN, USA Email（环境工程 明尼苏达大学 内华达州法戈 邮箱）；Environmental Engineering North Dakota State University Fargo, ND, USA Email（环境工程 内华达州立大学 内华达州法戈 邮箱）
In this paper, we presented a preliminary study for tactical driver behavior detection from untrimmed naturalistic driving recordings. While supervised learning based detection is a common approach, it suffers when labeled data is scarce. Manual annotation is both time-consuming and expensive. To emphasize this problem, we experimented on a 104-hour real-world naturalistic driving dataset with a set of predefined driving behaviors annotated. There are three challenges in the dataset. First, predefined driving behaviors are sparse in a naturalistic driving setting. Second, the distribution of d…
Universitas Indonesia（印度尼西亚大学）；Indonesian National Institute of Aeronautics and Space (LAPAN)（印度尼西亚国家航空航天研究所（LAPAN））
Deforestation is one of the contributing factors to climate change. Climate change has a serious impact on human life, and it occurs due to emission of greenhouse gases, such as carbon dioxide, to the atmosphere. It is important to know the causes of deforestation for mitigation efforts, but there is a lack of data-driven research studies to predict these deforestation drivers. In this work, we propose a contrastive learning architecture, called Multimodal SuperCon, for classifying drivers of deforestation in Indonesia using satellite images obtained from Landsat 8. Multimodal SuperCon is an a…
Distracted driving is deadly, claiming 3,477 lives in the U.S. in 2015 alone. Although there has been a considerable amount of research on modeling the distracted behavior of drivers under various conditions, accurate automatic detection using multiple modalities and especially the contribution of using the speech modality to improve accuracy has received little attention. This paper introduces a new multimodal dataset for distracted driving behavior and discusses automatic distraction detection using features from three modalities: facial expression, speech and car signals. Detailed multimoda…
Institute for Anthropomatics and Robotics（拟人学与机器人学研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Visual recognition inside the vehicle cabin leads to safer driving and more intuitive human-vehicle interaction but such systems face substantial obstacles as they need to capture different granularities of driver behaviour while dealing with highly limited body visibility and changing illumination. Multimodal recognition mitigates a number of such issues: prediction outcomes of different sensors complement each other due to different modality-specific strengths and weaknesses. While several late fusion methods have been considered in previously published frameworks, they constantly feature di…
BMW Group（宝马集团）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））
There is a growing interest in more intelligent natural user interaction with the car. Hand gestures and speech are already being applied for driver-car interaction. Moreover, multimodal approaches are also showing promise in the automotive industry. In this paper, we utilize deep learning for a multimodal fusion network for referencing objects outside the vehicle. We use features from gaze, head pose and finger pointing simultaneously to precisely predict the referenced objects in different car poses. We demonstrate the practical limitations of each modality when used for a natural form of re…
Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal architecture for in-vehicle vigilance estimation from Electroencephalogram and Electrooculogram. To enable the system to focus on the most salient parts of the learned multimodal representations, we propose an architecture composed of a capsule attention mechanism following a deep Long…
Advanced in-cabin sensing technologies, especially vision based approaches, have tremendously progressed user interaction inside the vehicle, paving the way for new applications of natural user interaction. Just as humans use multiple modes to communicate with each other, we follow an approach which is characterized by simultaneously using multiple modalities to achieve natural human-machine interaction for a specific task: pointing to or glancing towards objects inside as well as outside the vehicle for deictic references. By tracking the movements of eye-gaze, head and finger, we design a mu…
PhysDrive: A Multimodal Remote Physiological Measurement Dataset for In-vehicle Driver Monitoring
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Sichuan Agricultural University（四川农业大学）
MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception
Beijing Institute of Technology（北京理工大学）；Tsinghua University（清华大学）；HKUST(GZ)（香港科技大学（广州））；Beijing University of Chemical Technology（北京化工大学）；Beihang University（北京航空航天大学）
Honda Research Institute, USA（美国本田研究院）；Carnegie Mellon University（卡内基梅隆大学）
The Tactical Driver Behavior modeling problem requires understanding of driver actions in complicated urban scenarios from a rich multi modal signals including video, LiDAR and CAN bus data streams. However, the majority of deep learning research is focused either on learning the vehicle/environment state (sensor fusion) or the driver policy (from temporal data), but not both. Learning both tasks end-to-end offers the richest distillation of knowledge, but presents challenges in formulation and successful training. In this work, we propose promising first steps in this direction. Inspired by t…
UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception
Energy and Transportation Domain, Beijing Institute of Technology（能源与交通领域，北京理工大学）；State Key Laboratory of Intelligent Technology and Systems and Department of Computer Science and Technology, Tsinghua University（智能技术与系统国家重点实验室和清华大学计算机科学与技术系）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（机械与航空航天工程学院，南洋理工大学）；School of Transportation Science and Engineering and the State Key Lab of Intelligent Transportation System, Beihang University（交通运输科学与工程学院和智能交通系统国家重点实验室，北京航空航天大学）；Beijing University of Chemical Technology（北京化工大学）
VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Li Auto Inc.（Li汽车公司）；the School of Aeronautics and Astronautics, Xiamen University（厦门大学航空航天学院）；The Hong Kong University of Science and Technology（香港科技大学）
Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Making
Machine Intelligence, Interaction, and Imagination (Mi3) Laboratory（机器智能、交互与想象实验室）；Laboratory for Intelligent and Safe Automobiles (LISA)（智能与安全汽车实验室）；Johns Hopkins University（约翰霍普金斯大学）；Center for Medicinal Cannabis Research (CMCR)（医药大麻研究中心）
NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Driving
ARC Training Centre for Automated Vehicles in Rural and Remote Regions (AVR3)（澳大利亚农村及偏远地区自动驾驶车辆ARC培训中心）；Université Gustave Eiffel（Gustave Eiffel大学）；Queensland University of Technology (QUT)（昆士兰科技大学）
KITE – Toronto Rehabilitation Institute, University Health Network（大学健康网络KITE-多伦多康复研究所）；Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）
Distracted driving is one of the major reasons for vehicle accidents. Therefore, detecting distracted driving behaviors is of paramount importance to reduce the millions of deaths and injuries occurring worldwide. Distracted or anomalous driving behaviors are deviations from 'normal' driving that need to be identified correctly to alert the driver. However, these driving behaviors do not comprise one specific type of driving style and their distribution can be different during the training and test phases of a classifier. We formulate this problem as a supervised contrastive learning approach …
Spatial-Temporal Perception with Causal Inference for Naturalistic Driving Action Recognition
School of Mechanical Engineering, Nanjing University of Science and Technology（南京理工大学机械工程学院）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西交利物浦大学先进技术学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
V-SenseDrive: A Privacy-Preserving Road Video and In-Vehicle Sensor Fusion Framework for Road Safety & Driver Behaviour Modelling
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST)（电气工程与计算机科学学院（SEECS），国立科学与技术大学（NUST））
TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving
Faculty of Marine Science and Technology, Beijing Institute of Technology（海洋科学与技术学院，北京理工大学）；State Key Laboratory of Intelligent Technology and Systems and Department of Computer Science and Technology, Tsinghua University（智能技术与系统国家重点实验室和计算机科学与技术系，清华大学）；School of Engineering and Applied Science, Yale University（工程与应用科学学院，耶鲁大学）；University of Toronto（多伦多大学）；Beijing University of Chemical Technology（北京化工大学）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心 (DFKI)）
Over the past decades, the addition of hundreds of sensors to modern vehicles has led to an exponential increase in their capabilities. This allows for novel approaches to interaction with the vehicle that go beyond traditional touch-based and voice command approaches, such as emotion recognition, head rotation, eye gaze, and pointing gestures. Although gaze and pointing gestures have been used before for referencing objects inside and outside vehicles, the multimodal interaction and fusion of these gestures have so far not been extensively studied. We propose a novel learning-based multimodal…
Artificially intelligent perception is increasingly present in the lives of every one of us. Vehicles are no exception, (...) In the near future, pattern recognition will have an even stronger role in vehicles, as self-driving cars will require automated ways to understand what is happening around (and within) them and act accordingly. (...) This doctoral work focused on advancing in-vehicle sensing through the research of novel computer vision and pattern recognition methodologies for both biometrics and wellbeing monitoring. The main focus has been on electrocardiogram (ECG) biometrics, a tr…
