Video anomaly detection (VAD) has been extensively studied. However, research on egocentric traffic videos with dynamic scenes lacks large-scale benchmark datasets as well as effective evaluation metrics. This paper proposes traffic anomaly detection with a \textit{when-where-what} pipeline to detect, localize, and recognize anomalous events from egocentric videos. We introduce a new dataset called Detection of Traffic Anomaly (DoTA) containing 4,677 videos with temporal, spatial, and categorical annotations. A new spatial-temporal area under curve (STAUC) evaluation metric is proposed and used with DoTA. State-of-the-art methods are benchmarked for two VAD-related tasks.Experimental results show STAUC is an effective VAD metric. To our knowledge, DoTA is the largest traffic anomaly dataset to-date and is the first supporting traffic anomaly studies across when-where-what perspectives. Our code and dataset can be found in: https://github.com/MoonBlvd/Detection-of-Traffic-Anomaly
Prince Sultan University（苏丹王子大学）；INESC-TEC（波尔图理工学院高等工程研究所技术与科学创新研究所）；Polytechnic Institute of Porto（波尔图理工学院）
Aggressive driving (i.e., car drifting) is a dangerous behavior that puts human safety and life into a significant risk. This behavior is considered as an anomaly concerning the regular traffic in public transportation roads. Recent techniques in deep learning proposed new approaches for anomaly detection in different contexts such as pedestrian monitoring, street fighting, and threat detection. In this paper, we propose a new anomaly detection framework applied to the detection of aggressive driving behavior. Our contribution consists in the development of a 3D neural network architecture, ba…
Distracted drivers are more likely to fail to anticipate hazards, which result in car accidents. Therefore, detecting anomalies in drivers' actions (i.e., any action deviating from normal driving) contains the utmost importance to reduce driver-related accidents. However, there are unbounded many anomalous actions that a driver can do while driving, which leads to an 'open set recognition' problem. Accordingly, instead of recognizing a set of anomalous actions that are commonly defined by previous dataset providers, in this work, we propose a contrastive learning approach to learn a metric to …
Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly
Beijing University of Posts and Telecommunications（北京邮电大学）；Queen Mary University of London（伦敦玛丽女王大学）；Nanyang Technological University（南洋理工大学）；Yunnan University（云南大学）；Zhejiang University（浙江大学）；China Telecom Co., Ltd. Sichuan Branch（中国电信股份有限公司四川分公司）
Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly
National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications（北京邮电大学国家移动网络技术工程研究中心）；Terminus Technologies Co., Ltd.（奇点智源科技有限公司）；School of Computing and Communications, Lancaster University（兰卡斯特大学计算与通信学院）
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Honda Research Institute（本田研究所）
Anomaly driving detection is an important problem in advanced driver assistance systems (ADAS). It is important to identify potential hazard scenarios as early as possible to avoid potential accidents. This study proposes an unsupervised method to quantify driving anomalies using a conditional generative adversarial network (GAN). The approach predicts upcoming driving scenarios by conditioning the models on the previously observed signals. The system uses the difference of the output from the discriminator between the predicted and actual signals as a metric to quantify the anomaly degree of …
The inability of state-of-the-art semantic segmentation methods to detect anomaly instances hinders them from being deployed in safety-critical and complex applications, such as autonomous driving. Recent approaches have focused on either leveraging segmentation uncertainty to identify anomalous areas or re-synthesizing the image from the semantic label map to find dissimilarities with the input image. In this work, we demonstrate that these two methodologies contain complementary information and can be combined to produce robust predictions for anomaly segmentation. We present a pixel-wise an…
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；Singapore Management University（新加坡管理大学）
State-of-the-art (SOTA) anomaly segmentation approaches on complex urban driving scenes explore pixel-wise classification uncertainty learned from outlier exposure, or external reconstruction models. However, previous uncertainty approaches that directly associate high uncertainty to anomaly may sometimes lead to incorrect anomaly predictions, and external reconstruction models tend to be too inefficient for real-time self-driving embedded systems. In this paper, we propose a new anomaly segmentation method, named pixel-wise energy-biased abstention learning (PEBAL), that explores pixel-wise a…
Road accident can be triggered by wet road because it decreases skid resistance. To prevent the road accident, detecting road surface abnomality is highly useful. In this paper, we propose the deep learning based cost-effective real-time anomaly detection architecture, naming with non-compression auto-encoder (NCAE). The proposed architecture can reflect forward and backward causality of time series information via convolutional operation. Moreover, the above architecture shows higher anomaly detection performance of published anomaly detection model via experiments. We conclude that NCAE as a…
Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences（创新移动性研究所（IIMo），英戈尔施塔特应用技术大学）；Fraunhofer Institute for Transportation and Infrastructure Systems IVI（弗劳恩霍夫交通与基础设施系统研究所IVI）
Foreign substances on the road surface, such as rainwater or black ice, reduce the friction between the tire and the surface. The above situation will reduce the braking performance and make difficult to control the vehicle body posture. In that case, there is a possibility of property damage at least. In the worst case, personal damage will be occured. To avoid this problem, a road anomaly detection model is proposed based on vehicle driving noise. However, the prior proposal does not consider the extra noise, mixed with driving noise, and skipping calculations for moments without vehicle dri…
When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Department of Software and Microelectronics, Peking University（软件与微电子系，北京大学）；School of Electronic and Computer Engineering, Peking University（电子与计算机工程学院，北京大学）；State Key Laboratory of Virtual Reality Technology and Systems, SCSE, Beihang University（虚拟现实技术与系统国家重点实验室，北航软件学院）；Peng Cheng Laboratory（鹏城实验室）；Baidu Inc（百度公司）
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Great progress has been achieved in the community of autonomous driving in the past few years. As a safety-critical problem, however, anomaly detection is a huge hurdle towards a large-scale deployment of autonomous vehicles in the real world. While many approaches, such as uncertainty estimation or segmentation-based image resynthesis, are extremely promising, there is more to be explored. Especially inspired by works on anomaly detection based on image resynthesis, we propose a novel approach for anomaly detection through style transfer. We leverage generative models to map an image from its…
Wet weather makes water film over the road and that film causes lower friction between tire and road surface. When a vehicle passes the low-friction road, the accident can occur up to 35% higher frequency than a normal condition road. In order to prevent accidents as above, identifying the road condition in real-time is essential. Thus, we propose a convolutional auto-encoder-based anomaly detection model for taking both less computational resources and achieving higher anomaly detection performance. The proposed model adopts a non-compression method rather than a conventional bottleneck struc…
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑科学智能科技研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Wuppertal（伍珀塔尔大学）；Interdisciplinary Center for Machine Learning and Data Analytics（机器学习与数据分析跨学科中心）
Fraunhofer Center for Machine Learning（弗劳恩霍夫机器学习中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Fraunhofer Institute for Optronics, System Technologies and Image Exploitation IOSB（弗劳恩霍夫光电子、系统技术与图像利用研究所（IOSB））
Anomaly detection plays in many fields of research, along with the strongly related task of outlier detection, a very important role. Especially within the context of the automated analysis of video material recorded by surveillance cameras, abnormal situations can be of very different nature. For this purpose this work investigates Generative-Adversarial-Network-based methods (GAN) for anomaly detection related to surveillance applications. The focus is on the usage of static camera setups, since this kind of camera is one of the most often used and belongs to the lower price segment. In orde…
From a safety perspective, a machine learning method embedded in real-world applications is required to distinguish irregular situations. For this reason, there has been a growing interest in the anomaly detection (AD) task. Since we cannot observe abnormal samples for most of the cases, recent AD methods attempt to formulate it as a task of classifying whether the sample is normal or not. However, they potentially fail when the given normal samples are inherited from diverse semantic labels. To tackle this problem, we introduce a latent class-condition-based AD scenario. In addition, we propo…
Rutgers University（罗格斯大学）；Honda Research Institute（本田研究院）
Corner cases are the main bottlenecks when applying Artificial Intelligence (AI) systems to safety-critical applications. An AI system should be intelligent enough to detect such situations so that system developers can prepare for subsequent planning. In this paper, we propose semi-supervised anomaly detection considering the imbalance of normal situations. In particular, driving data consists of multiple positive/normal situations (e.g., right turn, going straight), some of which (e.g., U-turn) could be as rare as anomalous situations. Existing machine learning based anomaly detection approa…
National Taiwan University（台湾大学）；National Tsing Hua University（清华大学）
Anomaly awareness is an essential capability for safety-critical applications such as autonomous driving. While recent progress of robotics and computer vision has enabled anomaly detection for image classification, anomaly detection on semantic segmentation is less explored. Conventional anomaly-aware systems assuming other existing classes as out-of-distribution (pseudo-unknown) classes for training a model will result in two drawbacks. (1) Unknown classes, which applications need to cope with, might not actually exist during training time. (2) Model performance would strongly rely on the cl…
Accurately extracting driving events is the way to maximize computational efficiency and anomaly detection performance in the tire frictional nose-based anomaly detection task. This study proposes a concise and highly useful method for improving the precision of the event extraction that is hindered by extra noise such as wind noise, which is difficult to characterize clearly due to its randomness. The core of the proposed method is based on the identification of the road friction sound corresponding to the frequency of interest and removing the opposite characteristics with several frequency …
Aselsan Inc（阿瑟尔桑公司）；Middle East Technical University（中东技术大学）
In this paper, an LSTM autoencoder-based architecture is utilized for drowsiness detection with ResNet-34 as feature extractor. The problem is considered as anomaly detection for a single subject; therefore, only the normal driving representations are learned and it is expected that drowsiness representations, yielding higher reconstruction losses, are to be distinguished according to the knowledge of the network. In our study, the confidence levels of normal and anomaly clips are investigated through the methodology of label assignment such that training performance of LSTM autoencoder and in…
We present BiPOCO, a Bi-directional trajectory predictor with POse COnstraints, for detecting anomalous activities of pedestrians in videos. In contrast to prior work based on feature reconstruction, our work identifies pedestrian anomalous events by forecasting their future trajectories and comparing the predictions with their expectations. We introduce a set of novel compositional pose-based losses with our predictor and leverage prediction errors of each body joint for pedestrian anomaly detection. Experimental results show that our BiPOCO approach can detect pedestrian anomalous activities…
Inexpensive sensing and computation, as well as insurance innovations, have made smart dashboard cameras ubiquitous. Increasingly, simple model-driven computer vision algorithms focused on lane departures or safe following distances are finding their way into these devices. Unfortunately, the long-tailed distribution of road hazards means that these hand-crafted pipelines are inadequate for driver safety systems. We propose to apply data-driven anomaly detection ideas from deep learning to dashcam videos, which hold the promise of bridging this gap. Unfortunately, there exists almost no litera…
See Tomorrow, Act Today: Foresight-Driven Autonomous Driving
School of Data Science, Fudan University（复旦大学数据科学学院）；Shanghai Innovation Institute（上海创新研究院）；Imperial College London（伦敦帝国理工学院）；University of Surrey（萨里大学）
Research in visual anomaly detection draws much interest due to its applications in surveillance. Common datasets for evaluation are constructed using a stationary camera overlooking a region of interest. Previous research has shown promising results in detecting spatial as well as temporal anomalies in these settings. The advent of self-driving cars provides an opportunity to apply visual anomaly detection in a more dynamic application yet no dataset exists in this type of environment. This paper presents a new anomaly detection dataset - the Highway Traffic Anomaly (HTA) dataset - for the pr…
Teaching Vision-Language-Action Models What to See and Where to Look
School of Electronic Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；National College for Excellent Engineers, Beihang University（北京航空航天大学卓越工程师学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；DiDi（滴滴出行）；State Key Laboratory of Media Convergence and Communication, Communication University of China（中国传媒大学媒体融合与传播国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Cyber Science and Technology, Beihang University（北京航空航天大学网络安全科学与技术学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
Video anomalies detection is the intersection of anomaly detection and visual intelligence. It has commercial applications in surveillance, security, self-driving cars and crop monitoring. Videos can capture a variety of anomalies. Due to efforts needed to label training data, unsupervised approaches to train anomaly detection models for videos is more practical An autoencoder is a neural network that is trained to recreate its input using latent representation of input also called a bottleneck layer. Variational autoencoder uses distribution (mean and variance) as compared to latent vector as…
