In this paper we investigate a robust method to identify anomalies in complex scenes. This task is performed by evaluating the collective behavior by extracting the local binary patterns (LBP) and Laplacian of Gaussian (LoG) features. We fuse both features together which are exploited to train an MLP neural network during the training stage, and the anomaly is identified on the test samples. Considering the challenge of tracking individuals in dense crowded scenes due to multiple occlusions and clutter, in this paper we extract LBP and LoG features and use them as an approximate representation of the anomalous situation. These features well match the appearance of anomaly and their consistency, and accuracy is higher both in regular and irregular areas compared to other descriptors. In this paper, these features are exploited as input prior to train the neural network. The MLP neural network is subsequently explored to consider these features that can detect the anomalous situation. The experimental tests are conducted on a set of benchmark video sequences commonly used for anomaly situation detection.
Indian Institute of Engineering Science and Technology, Shibpur（印度希布尔工程科学技术学院）；Indian Institute of Space Science and Technology（印度空间科学技术学院）
We present an improved clustering based, unsupervised anomalous trajectory detection algorithm for crowded scenes. The proposed work is based on four major steps, namely, extraction of trajectories from crowded scene video, extraction of several features from these trajectories, independent mean-shift clustering and anomaly detection. First, the trajectories of all moving objects in a crowd are extracted using a multi feature video object tracker. These trajectories are then transformed into a set of feature spaces. Mean shift clustering is applied on these feature matrices to obtain distinct …
Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similarity
In this paper, we propose an accurate and real-time anomaly detection and localization in crowded scenes, and two descriptors for representing anomalous behavior in video are proposed. We consider a video as being a set of cubic patches. Based on the low likelihood of an anomaly occurrence, and the redundancy of structures in normal patches in videos, two (global and local) views are considered for modeling the video. Our algorithm has two components, for (1) representing the patches using local and global descriptors, and for (2) modeling the training patches using a new representation. We ha…
Situational Scene Graph for Structured Human-centric Situation Understanding
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Centre for Frontier AI Research, Agency for Science, Technology and Research（新加坡科技研究局前沿人工智能研究中心）；Institute of High-Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）
From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Huazhong University of Science and Technology（华中科技大学）；The Hong Kong Polytechnic University（香港理工大学）
Institute of Automation Chinese Academy of Sciences Beijing, China
Real-time moving object detection in unconstrained scenes is a difficult task due to dynamic background, changing foreground appearance and limited computational resource. In this paper, an optical flow based moving object detection framework is proposed to address this problem. We utilize homography matrixes to online construct a background model in the form of optical flow. When judging out moving foregrounds from scenes, a dual-mode judge mechanism is designed to heighten the system's adaptation to challenging situations. In experiment part, two evaluation metrics are redefined for more pro…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）
There is growing interest in artificial intelligence to build socially intelligent robots. This requires machines to have the ability to "read" people's emotions, motivations, and other factors that affect behavior. Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips. Each graph consists of several types of nodes, to capture who is present in the clip, their emotional and physical attributes, their relationships (i.e., parent/child), and the interactions between them. Most interactions …
Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）
Real-time motion detection in non-stationary scenes is a difficult task due to dynamic background, changing foreground appearance and limited computational resource. These challenges degrade the performance of the existing methods in practical applications. In this paper, an optical flow based framework is proposed to address this problem. By applying a novel strategy to utilize optical flow, we enable our method being free of model constructing, training or updating and can be performed efficiently. Besides, a dual judgment mechanism with adaptive intervals and adaptive thresholds is designed…
Uncovering Anomalous Events for Marine Environmental Monitoring via Visual Anomaly Detection
Computer Science Department, IT University of Copenhagen（丹麦哥本哈根IT大学计算机科学系）；Anemo Robotics ApS（丹麦Anemo机器人公司）；Visual Analysis and Perception Laboratory, Aalborg University（丹麦奥胡斯大学视觉分析与感知实验室）；Pioneer Centre for Artificial Intelligence, Denmark（丹麦先锋人工智能中心）
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）；Manipal Institute of Technology Bengaluru（马尼帕尔理工学院班加罗尔分校）
UAV based surveillance is gaining much interest worldwide due to its extensive applications in monitoring wildlife, urban planning, disaster management, campus security, etc. These videos are analyzed for strange/odd/anomalous patterns which are essential aspects of surveillance. But manual analysis of these videos is tedious and laborious. Hence, the development of computer-aided systems for the analysis of UAV based surveillance videos is crucial. Despite this interest, in literature, several computer aided systems are developed focusing only on CCTV based surveillance videos. These methods …
Institute of Advanced Research in Artificial Intelligence（人工智能高级研究院）；Helmholtz-Zentrum Dresden-Rossendorf（亥姆霍兹德累斯顿罗森多夫研究中心）；Helmholtz Institute Freiberg for Resource Technology（亥姆霍兹弗赖贝格资源技术研究所）
In recent years, multi-view subspace learning has been garnering increasing attention. It aims to capture the inner relationships of the data that are collected from multiple sources by learning a unified representation. In this way, comprehensive information from multiple views is shared and preserved for the generalization processes. As a special branch of temporal series hyperspectral image (HSI) processing, the anomalous change detection task focuses on detecting very small changes among different temporal images. However, when the volume of datasets is very large or the classes are relati…
Avoiding bottleneck situations in crowds is critical for the safety and comfort of people at large events or in public transportation. Based on the work of Lagrangian motion analysis we propose a novel video-based bottleneckdetector by identifying characteristic stowage patterns in crowd-movements captured by optical flow fields. The Lagrangian framework allows to assess complex timedependent crowd-motion dynamics at large temporal scales near the bottleneck by two dimensional Lagrangian fields. In particular we propose long-term temporal filtered Finite Time Lyapunov Exponents (FTLE) fields t…
In a recorded situation, textual information is crucial for scene interpretation and decision making. The ability to edit text directly on images has a number of advantages, including error correction, text restoration, and image reusability. This research shows how to change image text at the letter and digits level. I devised a two-part letters-digits network (LDN) to encode and decode digital images, as well as learn and transfer the font style of the source characters to the target characters. This method allows you to update the uppercase letters, lowercase letters and digits in the pictu…
Lane detection in driving scenes is an important module for autonomous vehicles and advanced driver assistance systems. In recent years, many sophisticated lane detection methods have been proposed. However, most methods focus on detecting the lane from one single image, and often lead to unsatisfactory performance in handling some extremely-bad situations such as heavy shadow, severe mark degradation, serious vehicle occlusion, and so on. In fact, lanes are continuous line structures on the road. Consequently, the lane that cannot be accurately detected in one current frame may potentially be…
COMSATS University Islamabad, Wah Campus（伊斯兰堡康萨茨大学瓦赫校区）
Human actions that do not conform to usual behavior are considered as anomalous and such actors are called anomalous entities. Detection of anomalous entities using visual data is a challenging problem in computer vision. This paper presents a new approach to detect anomalous entities in complex situations of examination halls. The proposed method uses a cascade of deep convolutional neural network models. In the first stage, we apply a pretrained model of human pose estimation on frames of videos to extract key feature points of body. Patches extracted from each key point are utilized in the …
Southeast University（东南大学）；Centre de Recherche en Information Biomédicale Sino-Français (CRIBs)（中法生物医学信息研究中心）；Univ Rennes（雷恩大学）；Inserm（法国国家健康与医学研究院）；LTSI - UMR1099（信号与图像处理实验室）
Deep learning-based medical image registration and segmentation joint models utilize the complementarity (augmentation data or weakly supervised data from registration, region constraints from segmentation) to bring mutual improvement in complex scene and few-shot situation. However, further adoption of the joint models are hindered: 1) the diversity of augmentation data is reduced limiting the further enhancement of segmentation, 2) misaligned regions in weakly supervised data disturb the training process, 3) lack of label-based region constraints in few-shot situation limits the registration…
Rutgers University（罗格斯大学）；Honda Research Institute（本田研究院）
Corner cases are the main bottlenecks when applying Artificial Intelligence (AI) systems to safety-critical applications. An AI system should be intelligent enough to detect such situations so that system developers can prepare for subsequent planning. In this paper, we propose semi-supervised anomaly detection considering the imbalance of normal situations. In particular, driving data consists of multiple positive/normal situations (e.g., right turn, going straight), some of which (e.g., U-turn) could be as rare as anomalous situations. Existing machine learning based anomaly detection approa…
Existing datasets for training pedestrian detectors in images suffer from limited appearance and pose variation. The most challenging scenarios are rarely included because they are too difficult to capture due to safety reasons, or they are very unlikely to happen. The strict safety requirements in assisted and autonomous driving applications call for an extra high detection accuracy also in these rare situations. Having the ability to generate people images in arbitrary poses, with arbitrary appearances and embedded in different background scenes with varying illumination and weather conditio…
Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；Indian Statistical Institute Calcutta（印度统计研究所加尔各答分所）
Haze and fog reduce the visibility of outdoor scenes as a veil like semi-transparent layer appears over the objects. As a result, images captured under such conditions lack contrast. Image dehazing methods try to alleviate this problem by recovering a clear version of the image. In this paper, we propose a Fully Convolutional Neural Network based model to recover the clear scene radiance by estimating the environmental illumination and the scene transmittance jointly from a hazy image. The method uses a relaxed haze imaging model to allow for the situations with non-uniform illumination. We ha…
Xi’an Institute of Optics and Precision Mechanics of CAS（中国科学院西安光学精密机械研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Fraunhofer IGD（弗劳恩霍夫计算机图形研究所）
3D feature descriptor provide information between corresponding models and scenes. 3D objection recognition in cluttered scenes, however, remains a largely unsolved problem. Practical applications impose several challenges which are not fully addressed by existing methods. Especially in cluttered scenes there are many feature mismatches between scenes and models. We therefore propose Histograms of Gaussian Normal Distribution (HGND) for extracting salient features on a local reference frame (LRF) that enables us to solve this problem. We propose a LRF on each local surface patches using the sc…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
We develop a novel framework for single-scene video anomaly localization that allows for human-understandable reasons for the decisions the system makes. We first learn general representations of objects and their motions (using deep networks) and then use these representations to build a high-level, location-dependent model of any particular scene. This model can be used to detect anomalies in new videos of the same scene. Importantly, our approach is explainable - our high-level appearance and motion features can provide human-understandable reasons for why any part of a video is classified …
OoDDINO:A Multi-level Framework for Anomaly Segmentation on Complex Road Scenes
College of Computer Science and Artificial Intelligence, Southwest Minzu University（计算机科学与人工智能学院，西南民族大学）；Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, China（可持续城市智能交通工程研究中心，教育部，中国）
School of Computer and Artificial Intelligence, Zhengzhou University（郑州大学计算机与人工智能学院）；Henan Institute of Advanced Technology, Zhengzhou University（郑州大学河南先进技术研究院）；NLPR, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Pedestrian trajectory prediction is an important technique of autonomous driving, which has become a research hot-spot in recent years. Previous methods mainly rely on the position relationship of pedestrians to model social interaction, which is obviously not enough to represent the complex cases in real situations. In addition, most of existing work usually introduce the scene interaction module as an independent branch and embed the social interaction features in the process of trajectory generation, rather than simultaneously carrying out the social interaction and scene interaction, which…
Beihang University（北京航空航天大学）；Dalian Maritime University（大连海事大学）；University of Technology of Troyes（特鲁瓦工程技术大学）
It is challenging to detect the anomaly in crowded scenes for quite a long time. In this paper, a self-supervised framework, abnormal event detection network (AED-Net), which is composed of PCAnet and kernel principal component analysis (kPCA), is proposed to address this problem. Using surveillance video sequences of different scenes as raw data, PCAnet is trained to extract high-level semantics of crowd's situation. Next, kPCA,a one-class classifier, is trained to determine anomaly of the scene. In contrast to some prevailing deep learning methods,the framework is completely self-supervised …
Gwangju Institute of Science and Technology（光州科学技术院）
In this paper, we propose the methods to handle temporal errors during multi-object tracking. Temporal error occurs when objects are occluded or noisy detections appear near the object. In those situations, tracking may fail and various errors like drift or ID-switching occur. It is hard to overcome temporal errors only by using motion and shape information. So, we propose the historical appearance matching method and joint-input siamese network which was trained by 2-step process. It can prevent tracking failures although objects are temporally occluded or last matching information is unrelia…
Studies on microscopic pedestrian requires large amounts of trajectory data from real-world pedestrian crowds. Such data collection, if done manually, needs tremendous effort and is very time consuming. Though many studies have asserted the possibility of automating this task using video cameras, we found that only a few have demonstrated good performance in very crowded situations or from a top-angled view scene. This paper deals with tracking pedestrian crowd under heavy occlusions from an angular scene. Our automated tracking system consists of two modules that perform sequentially. The fir…
University of Trento（特伦托大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）；University of Perugia（佩鲁贾大学）；ADSC, UIUC（伊利诺伊大学厄巴纳-香槟分校高级数字科学中心）
We present a novel unsupervised deep learning framework for anomalous event detection in complex video scenes. While most existing works merely use hand-crafted appearance and motion features, we propose Appearance and Motion DeepNet (AMDN) which utilizes deep neural networks to automatically learn feature representations. To exploit the complementary information of both appearance and motion patterns, we introduce a novel double fusion framework, combining both the benefits of traditional early fusion and late fusion strategies. Specifically, stacked denoising autoencoders are proposed to sep…
The inability of state-of-the-art semantic segmentation methods to detect anomaly instances hinders them from being deployed in safety-critical and complex applications, such as autonomous driving. Recent approaches have focused on either leveraging segmentation uncertainty to identify anomalous areas or re-synthesizing the image from the semantic label map to find dissimilarities with the input image. In this work, we demonstrate that these two methodologies contain complementary information and can be combined to produce robust predictions for anomaly segmentation. We present a pixel-wise an…
In recent years, many works have addressed the problem of finding never-seen-before anomalies in videos. Yet, most work has been focused on detecting anomalous frames in surveillance videos taken from security cameras. Meanwhile, the task of anomaly detection (AD) in videos exhibiting anomalous mechanical behavior, has been mostly overlooked. Anomaly detection in such videos is both of academic and practical interest, as they may enable automatic detection of malfunctions in many manufacturing, maintenance, and real-life settings. To assess the potential of the different approaches to detect s…
Traffic Management Centers (TMCs) routinely use traffic cameras to provide situational awareness regarding traffic, road, and weather conditions. Camera footage is quite useful for a variety of diagnostic purposes; yet, most footage is kept for only a few days, if at all. This is largely due to the fact that currently, identification of notable footage is done via manual review by human operators---a laborious and inefficient process. In this article, we propose a semantics-oriented approach to analyzing sequential image data, and demonstrate its application for automatic detection of real-wor…
Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we…
Gwangju Institute of Science and Technology (GIST)（光州科学技术院）；Mokwon University（牧园大学）
We propose a condition-adaptive representation learning framework for the driver drowsiness detection based on 3D-deep convolutional neural network. The proposed framework consists of four models: spatio-temporal representation learning, scene condition understanding, feature fusion, and drowsiness detection. The spatio-temporal representation learning extracts features that can describe motions and appearances in video simultaneously. The scene condition understanding classifies the scene conditions related to various conditions about the drivers and driving situations such as statuses of wea…
Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）；Point Loma Nazarene University（波因特洛马拿撒勒大学）；NOAA（美国国家海洋和大气管理局）
This paper explores the use of a Bayesian non-parametric topic modeling technique for the purpose of anomaly detection in video data. We present results from two experiments. The first experiment shows that the proposed technique is automatically able characterize the underlying terrain, and detect anomalous flora in image data collected by an underwater robot. The second experiment shows that the same technique can be used on images from a static camera in a dynamic unstructured environment. In the second dataset, consisting of video data from a static seafloor camera capturing images of a bu…
Universitat de València（瓦伦西亚大学）；Descartes Lab（笛卡尔实验室）
Anomalous change detection (ACD) is an important problem in remote sensing image processing. Detecting not only pervasive but also anomalous or extreme changes has many applications for which methodologies are available. This paper introduces a nonlinear extension of a full family of anomalous change detectors. In particular, we focus on algorithms that utilize Gaussian and elliptically contoured (EC) distribution and extend them to their nonlinear counterparts based on the theory of reproducing kernels' Hilbert space. We illustrate the performance of the kernel methods introduced in both perv…
To assist human drivers and autonomous vehicles in assessing crash risks, driving scene analysis using dash cameras on vehicles and deep learning algorithms is of paramount importance. Although these technologies are increasingly available, driving scene analysis for this purpose still remains a challenge. This is mainly due to the lack of annotated large image datasets for analyzing crash risk indicators and crash likelihood, and the lack of an effective method to extract lots of required information from complex driving scenes. To fill the gap, this paper develops a scene analysis system. Th…
We present Affect2MM, a learning method for time-series emotion prediction for multimedia content. Our goal is to automatically capture the varying emotions depicted by characters in real-life human-centric situations and behaviors. We use the ideas from emotion causation theories to computationally model and determine the emotional state evoked in clips of movies. Affect2MM explicitly models the temporal causality using attention-based methods and Granger causality. We use a variety of components like facial features of actors involved, scene understanding, visual aesthetics, action/situation…
Normandie Univ（诺曼底大学）；INSA Rouen（鲁昂国立应用科学学院）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；LITIS（LITIS实验室）；University of Burgundy（勃艮第大学）；UBFC（勃艮第-弗朗什孔泰大学）；ImViA（ImViA实验室）
Object detection in road scenes is necessary to develop both autonomous vehicles and driving assistance systems. Even if deep neural networks for recognition task have shown great performances using conventional images, they fail to detect objects in road scenes in complex acquisition situations. In contrast, polarization images, characterizing the light wave, can robustly describe important physical properties of the object even under poor illumination or strong reflections. This paper shows how non-conventional polarimetric imaging modality overcomes the classical methods for object detectio…
Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）
Humans demonstrate remarkable abilities to predict physical events in complex scenes. Two classes of models for physical scene understanding have recently been proposed: "Intuitive Physics Engines", or IPEs, which posit that people make predictions by running approximate probabilistic simulations in causal mental models similar in nature to video-game physics engines, and memory-based models, which make judgments based on analogies to stored experiences of previously encountered scenes and physical outcomes. Versions of the latter have recently been instantiated in convolutional neural network…
