Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding
Carnegie Mellon University（卡内基梅隆大学）；University of Trento（特伦托大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Aaron Lohner, Francesco Compagno, Jonathan Francis, Alessandro Oltramari
备注：
Won the 'Best Paper Runner-up Award' at the 2024 IEEE International Automated Vehicle Validation Conference (IAVVC 2024). Also accepted at the 1st Workshop on Semantic Reasoning and Goal Understanding in Robotics, at the Robotics Science and Systems Conference (RSS SemRob 2024)
识别交通事故是任何自动驾驶或道路监控系统的重要组成部分。事故可能以多种多样的形式出现，而理解正在发生的事故类型可能有助于防止其再次发生。本工作聚焦于将交通场景分类为具体的事故类型。我们通过把交通场景表示为图来处理该问题，其中汽车等对象可表示为节点，它们之间的相对距离和方向表示为边。这种交通场景表示被称为场景图，并可用作事故分类器的输入。将场景图输入与视觉和文本表示相融合的分类器能够获得更好的结果。本工作提出了一种多阶段、多模态流水线，对交通事故视频进行预处理，将其编码为场景图，并在执行分类任务之前将该表示与视觉和语言模态对齐。当在4个类别上训练时，我们的方法在流行的 Detection of Traffic Anomaly（DoTA，交通异常检测）基准的一个（不平衡）子集上取得了57.77%的平衡准确率，相比未考虑场景图信息的情况提高了近5个百分点。
英文摘要：
Recognizing a traffic accident is an essential part of any autonomous driving or road monitoring system. An accident can appear in a wide variety of forms, and understanding what type of accident is taking place may be useful to prevent it from recurring. This work focuses on classifying traffic scenes into specific accident types. We approach the problem by representing a traffic scene as a graph, where objects such as cars can be represented as nodes, and relative distances and directions between them as edges. This representation of a traffic scene is referred to as a scene graph, and can be used as input for an accident classifier. Better results are obtained with a classifier that fuses the scene graph input with visual and textual representations. This work introduces a multi-stage, multimodal pipeline that pre-processes videos of traffic accidents, encodes them as scene graphs, and aligns this representation with vision and language modalities before executing the classification task. When trained on 4 classes, our method achieves a balanced accuracy score of 57.77% on an (unbalanced) subset of the popular Detection of Traffic Anomaly (DoTA) benchmark, representing an increase of close to 5 percentage points from the case where scene graph information is not taken into account.
Surveillance Video-Based Traffic Accident Detection Using Transformer Architecture
Department of Civil Engineering, Indian Institute of Technology Kanpur（印度理工学院坎浦尔分校土木工程系）；School of Computing, Engineering and Mathematical Sciences, La Trobe University（拉特罗布大学计算科学与工程数学科学学院）
Stanford University（斯坦福大学）；NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, these scenarios must be realistic and challenging, but not impossible to drive through safely. In this work, we introduce STRIVE, a method to automatically generate challenging scenarios that cause a given planner to produce undesirable behavior, like collisions. To maintain scenario plausibility, the key idea is to leverage a learned model of traffic motion in the form of a graph-based conditional VAE. Scenario generation is formulated as an optimization in the …
Traffic Incident Database with Multiple Labels Including Various Perspective Environmental Information
Aichi Institute of Technology（爱知工业大学）；Tokyo Denki University（东京电机大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所（AIST））；Fukuoka University（福冈大学）；Keio University（庆应义塾大学）
Recently, the problem of traffic accident risk forecasting has been getting the attention of the intelligent transportation systems community due to its significant impact on traffic clearance. This problem is commonly tackled in the literature by using data-driven approaches that model the spatial and temporal incident impact, since they were shown to be crucial for the traffic accident risk forecasting problem. To achieve this, most approaches build different architectures to capture the spatio-temporal correlations features, making them inefficient for large traffic accident datasets. Thus,…
EQ-TAA: Equivariant Traffic Accident Anticipation via Diffusion-Based Accident Video Synthesis
Xi’an Jiaotong University（西安交通大学）；University of Macau（澳门大学）；National University of Singapore（新加坡国立大学）；Department of Electrical Engineering and Computer Science, Cleveland State University（克莱姆森大学电气工程与计算机科学系）；Institute for Infocomm Research, Agency for Science, Technology and Research (A ∗ STAR)（信息与通信研究机构，科技研究局）
New Jersey Institute of Technology（新泽西理工学院）；Innovative AI Technologies（创新人工智能技术公司）
Automatic detection of traffic accidents is an important emerging topic in traffic monitoring systems. Nowadays many urban intersections are equipped with surveillance cameras connected to traffic management systems. Therefore, computer vision techniques can be viable tools for automatic accident detection. This paper presents a new efficient framework for accident detection at intersections for traffic surveillance applications. The proposed framework consists of three hierarchical steps, including efficient and accurate object detection based on the state-of-the-art YOLOv4 method, object tra…
The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipati…
AI Innovation and Application Center of China Unicom（中国联通人工智能创新应用中心）
Automatic traffic accidents detection has appealed to the machine vision community due to its implications on the development of autonomous intelligent transportation systems (ITS) and importance to traffic safety. Most previous studies on efficient analysis and prediction of traffic accidents, however, have used small-scale datasets with limited coverage, which limits their effect and applicability. Existing datasets in traffic accidents are either small-scale, not from surveillance cameras, not open-sourced, or not built for freeway scenes. Since accidents happened in freeways tend to cause …
Traffic accident anticipation aims to predict accidents from dashcam videos as early as possible, which is critical to safety-guaranteed self-driving systems. With cluttered traffic scenes and limited visual cues, it is of great challenge to predict how long there will be an accident from early observed frames. Most existing approaches are developed to learn features of accident-relevant agents for accident anticipation, while ignoring the features of their spatial and temporal relations. Besides, current deterministic deep neural networks could be overconfident in false predictions, leading t…
Indiana University（印第安纳大学）；University of Michigan（密歇根大学）
Recognizing abnormal events such as traffic violations and accidents in natural driving scenes is essential for successful autonomous driving and advanced driver assistance systems. However, most work on video anomaly detection suffers from two crucial drawbacks. First, they assume cameras are fixed and videos have static backgrounds, which is reasonable for surveillance applications but not for vehicle-mounted cameras. Second, they pose the problem as one-class classification, relying on arduously hand-labeled training datasets that limit recognition to anomaly categories that have been expli…
Learning physically grounded traffic accident reconstruction from public accident reports
State Key Laboratory of Internet of Things for Smart City（物联网智能城市国家重点实验室）；Department of Civil Engineering（土木工程系）；Department of Computer and Information Science（计算机与信息科学系）
Big Data Integration Research Center（大数据集成研究中心）；NICT（日本信息通信研究机构）
To develop the advanced self-driving systems, many researchers are focusing to alert all possible traffic risk cases from closed-circuit television (CCTV) and dashboard-mounted cameras. Most of these methods focused on identifying frame-by-frame in which an anomaly has occurred, but they are unrealized, which road traffic participant can cause ego-vehicle leading into collision because of available annotation dataset only to detect anomaly on traffic video. Near-miss is one type of accident and can be defined as a narrowly avoided accident. However, there is no difference between accident and …
University of Cincinnati（辛辛那提大学）；Central Michigan University（中央密歇根大学）；University of Louisiana at Lafayette（路易斯安那大学拉斐特分校）
Action detection and public traffic safety are crucial aspects of a safe community and a better society. Monitoring traffic flows in a smart city using different surveillance cameras can play a significant role in recognizing accidents and alerting first responders. The utilization of action recognition (AR) in computer vision tasks has contributed towards high-precision applications in video surveillance, medical imaging, and digital signal processing. This paper presents an intensive review focusing on action recognition in accident detection and autonomous transportation systems for a smart…
In Intelligent Transportation System, real-time systems that monitor and analyze road users become increasingly critical as we march toward the smart city era. Vision-based frameworks for Object Detection, Multiple Object Tracking, and Traffic Near Accident Detection are important applications of Intelligent Transportation System, particularly in video surveillance and etc. Although deep neural networks have recently achieved great success in many computer vision tasks, a uniformed framework for all the three tasks is still challenging where the challenges multiply from demand for real-time pe…
National Institute of Technology Warangal（瓦朗加尔国家理工学院）
Computer vision-based accident detection through video surveillance has become a beneficial but daunting task. In this paper, a neoteric framework for detection of road accidents is proposed. The proposed framework capitalizes on Mask R-CNN for accurate object detection followed by an efficient centroid based object tracking algorithm for surveillance footage. The probability of an accident is determined based on speed and trajectory anomalies in a vehicle after an overlap with other vehicles. The proposed framework provides a robust method to achieve a high Detection Rate and a low False Alar…
Traffic accident anticipation is a vital function of Automated Driving Systems (ADSs) for providing a safety-guaranteed driving experience. An accident anticipation model aims to predict accidents promptly and accurately before they occur. Existing Artificial Intelligence (AI) models of accident anticipation lack a human-interpretable explanation of their decision-making. Although these models perform well, they remain a black-box to the ADS users, thus difficult to get their trust. To this end, this paper presents a Gated Recurrent Unit (GRU) network that learns spatio-temporal relational fea…
Causal-Entity Reflected Egocentric Traffic Accident Video Synthesis
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（国立新加坡大学）；Nanyang Technological University（南洋理工大学）；Cleveland State University（克利夫兰州立大学）
National Institute of Advanced Industrial Science and Technology (AIST)（国立先进工业科学技术研究所（AIST））；Keio University（庆应义塾大学）
In this paper, we propose a novel approach for traffic accident anticipation through (i) Adaptive Loss for Early Anticipation (AdaLEA) and (ii) a large-scale self-annotated incident database for anticipation. The proposed AdaLEA allows a model to gradually learn an earlier anticipation as training progresses. The loss function adaptively assigns penalty weights depending on how early the model can an- ticipate a traffic accident at each epoch. Additionally, we construct a Near-miss Incident DataBase for anticipation. This database contains an enormous number of traffic near- miss incident vide…
Safety-Critical Learning for Long-Tail Events: The TUM Traffic Accident Dataset
Technical University of Munich（慕尼黑技术大学）；Fraunhofer Institute for Transportation and Infrastructure Systems（交通与基础设施系统弗劳恩霍夫研究所）；SETLabs Research GmbH（SETLabs研究公司）；University of California Merced（加州大学默塞德分校）
The robustness of semantic segmentation on edge cases of traffic scene is a vital factor for the safety of intelligent transportation. However, most of the critical scenes of traffic accidents are extremely dynamic and previously unseen, which seriously harm the performance of semantic segmentation methods. In addition, the delay of the traditional camera during high-speed driving will further reduce the contextual information in the time dimension. Therefore, we propose to extract dynamic context from event-based data with a higher temporal resolution to enhance static RGB images, even for th…
Carnegie Mellon University（卡内基梅隆大学）；University of Tokyo（东京大学）
This paper presents a novel dataset for traffic accidents analysis. Our goal is to resolve the lack of public data for research about automatic spatio-temporal annotations for traffic safety in the roads. Through the analysis of the proposed dataset, we observed a significant degradation of object detection in pedestrian category in our dataset, due to the object sizes and complexity of the scenes. To this end, we propose to integrate contextual information into conventional Faster R-CNN using Context Mining (CM) and Augmented Context Mining (ACM) to complement the accuracy for small pedestria…
Fudan University（复旦大学）；China University of Petroleum（中国石油大学）
Predicting risk map of traffic accidents is vital for accident prevention and early planning of emergency response. Here, the challenge lies in the multimodal nature of urban big data. We propose a compact neural ensemble model to alleviate overfitting in fusing multimodal features and develop some new features such as fractal measure of road complexity in satellite images, taxi flows, POIs, and road width and connectivity in OpenStreetMap. The solution is more promising in performance than the baseline methods and the single-modality data based solutions. After visualization from a micro view…
Deep Learning Advances in Vision-Based Traffic Accident Anticipation: A Comprehensive Review of Methods, Datasets, and Future Directions
School of Advanced Manufacturing, Fuzhou University（福州大学先进制造学院）；National Local United Engineering Lab of Flat Panel Display Technology, Fuzhou University（福州大学平板显示技术国家地方联合工程实验室）；College of Computer Software, Hanyang University（汉阳大学计算机软件学院）；College of Computing, City University of Hong Kong（香港城市大学计算学院）
Universidad Nacional de San Antonio Abad del Cusco（库斯科圣安东尼奥阿巴德国立大学）；Pontificia Universidad Católica del Perú（秘鲁天主教大学）
The development of autonomous vehicles arises new challenges in urban traffic scenarios where vehicle-pedestrian interactions are frequent e.g. vehicle yields to pedestrians, pedestrian slows down due approaching to the vehicle. Over the last years, several datasets have been developed to model these interactions. However, available datasets do not cover near-accident scenarios that our dataset covers. We introduce I see you, a new vehicle-pedestrian interaction dataset that tackles the lack of trajectory data in near-accident scenarios using YOLOv5 and camera calibration methods. I see you co…
Investigating Traffic Accident Detection Using Multimodal Large Language Models
Embedded Systems Group (Dept.-E)（嵌入式系统组）；Virtual Vehicle Research GmbH（虚拟车辆研究公司）；Control Systems Group (Dept.-E)（控制系统组）；Institute of Visual Computing（视觉计算研究所）；Graz University of Technology（格拉茨技术大学）
Ensuring the safety of all traffic participants is a prerequisite for bringing intelligent vehicles closer to practical applications. The assistance system should not only achieve high accuracy under normal conditions, but obtain robust perception against extreme situations. However, traffic accidents that involve object collisions, deformations, overturns, etc., yet unseen in most training sets, will largely harm the performance of existing semantic segmentation models. To tackle this issue, we present a rarely addressed task regarding semantic segmentation in accidental scenarios, along with…
Real-time Accident Anticipation for Autonomous Driving Through Monocular Depth-Enhanced 3D Modeling
University of Macau（澳门大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；New York University（纽约大学）；Central South University（中南大学）；Beihang University（北京航空航天大学）；University of Hawaii（夏威夷大学）
Traffic accident anticipation aims to accurately and promptly predict the occurrence of a future accident from dashcam videos, which is vital for a safety-guaranteed self-driving system. To encourage an early and accurate decision, existing approaches typically focus on capturing the cues of spatial and temporal context before a future accident occurs. However, their decision-making lacks visual explanation and ignores the dynamic interaction with the environment. In this paper, we propose Deep ReInforced accident anticipation with Visual Explanation, named DRIVE. The method simulates both the…
Enhancing Highway Safety: Accident Detection on the A9 Test Stretch Using Roadside Sensors
Technical University of Munich (TUM)（慕尼黑工业大学）；University of California Merced (UCM)（加利福尼亚大学默塞德分校）；Fraunhofer Institute for Transportation and Infrastructure Systems (IVI)（弗劳恩霍夫交通与基础设施系统研究所）；SETLabs Research GmbH（SETLabs 研究有限公司）
The sun glare is one of the major environmental hazards that cause traffic accidents. Every year, many people died and injured in traffic accidents related to sun glare. Providing accurate information about when and where sun glare happens would be helpful to prevent sun glare caused traffic accidents and save lives. In this study, we proposed to use publicly accessible Google Street View (GSV) panorama images to estimate and predict the occurrence of sun glare. GSV images have view sight similar to drivers, which would make GSV images suitable for estimating the visibility of sun glare to dri…
