This work introduces a new approach to localize anomalies in surveillance video. The main novelty is the idea of using a Siamese convolutional neural network (CNN) to learn a distance function between a pair of video patches (spatio-temporal regions of video). The learned distance function, which is not specific to the target video, is used to measure the distance between each video patch in the testing video and the video patches found in normal training video. If a testing video patch is not similar to any normal video patch then it must be anomalous. We compare our approach to previously published algorithms using 4 evaluation measures and 3 challenging target benchmark datasets. Experiments show that our approach either surpasses or performs comparably to current state-of-the-art methods.
Deep Learning methods have been extensively used to analyze video data to extract valuable information by classifying image frames and detecting objects. We describe a unique approach for using video feed from a moving Locomotive to continuously monitor the Railway Track and detect significant assets like Switches on the Track. The technique used here is called Siamese Networks, which uses 2 identical networks to learn the similarity between of 2 images. Here we will use a Siamese network to continuously compare Track images and detect any significant difference in the Track. Switch will be on…
Indian Institute of Science（印度科学学院）；Robert Bosch Center for Cyber-Physical Systems（罗伯特·博世信息物理系统研究中心）
A robot that can operate in novel and unstructured environments must be capable of recognizing new, previously unseen, objects. In this work, a visual cue is used to specify a novel object of interest which must be localized in new environments. An end-to-end neural network equipped with a Siamese network is used to learn the cue, infer the object of interest, and then to localize it in new environments. We show that a simulated robot can pick-and-place novel objects pointed to by a laser pointer. We also evaluate the performance of the proposed approach on a dataset derived from the Omniglot …
Laboratoire d’imagerie, de vision et d’intelligence artificielle (LIVIA), Ecole de technologie superieure（高等技术学院 成像、视觉与人工智能实验室（LIVIA））；Vellore Institute of Technology（韦洛尔理工学院）；Genetec Inc.（Genetec公司）
Siamese trackers perform similarity matching with templates (i.e., target models) to recursively localize objects within a search region. Several strategies have been proposed in the literature to update a template based on the tracker output, typically extracted from the target search region in the current frame, and thereby mitigate the effects of target drift. However, this may lead to corrupted templates, limiting the potential benefits of a template update strategy. This paper proposes a model adaptation method for Siamese trackers that uses a generative model to produce a synthetic templ…
Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；ETH Zürich（苏黎世联邦理工学院）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）；Linköping University（林雪平大学）
Siamese approaches address the visual tracking problem by extracting an appearance template from the current frame, which is used to localize the target in the next frame. In general, this template is linearly combined with the accumulated template from the previous frame, resulting in an exponential decay of information over time. While such an approach to updating has led to improved results, its simplicity limits the potential gain likely to be obtained by learning to update. Therefore, we propose to replace the handcrafted update function with a method which learns to update. We use a conv…
Westlake University（西湖大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；The future network of intelligence institute (FNII)（未来网络智能研究院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
3D single object tracking (3D SOT) in LiDAR point clouds plays a crucial role in autonomous driving. Current approaches all follow the Siamese paradigm based on appearance matching. However, LiDAR point clouds are usually textureless and incomplete, which hinders effective appearance matching. Besides, previous methods greatly overlook the critical motion clues among targets. In this work, beyond 3D Siamese tracking, we introduce a motion-centric paradigm to handle 3D SOT from a new perspective. Following this paradigm, we propose a matching-free two-stage tracker M^2-Track. At the 1^st-stage,…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）；JD Finance America Corporation（京东金融美国公司）
In this paper, we present a novel siamese motion-aware network (SiamMan) for visual tracking, which consists of the siamese feature extraction subnetwork, followed by the classification, regression, and localization branches in parallel. The classification branch is used to distinguish the foreground from background, and the regression branch is adopt to regress the bounding box of target. To reduce the impact of manually designed anchor boxes to adapt to different target motion patterns, we design the localization branch, which aims to coarsely localize the target to help the regression branc…
Image-guided radiation therapy can benefit from accurate motion tracking by ultrasound imaging, in order to minimize treatment margins and radiate moving anatomical targets, e.g., due to breathing. One way to formulate this tracking problem is the automatic localization of given tracked anatomical landmarks throughout a temporal ultrasound sequence. For this, we herein propose a fully-convolutional Siamese network that learns the similarity between pairs of image regions containing the same landmark. Accordingly, it learns to localize and thus track arbitrary image features, not only predefine…
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）
Weakly supervised detection of anomalies in surveillance videos is a challenging task. Going beyond existing works that have deficient capabilities to localize anomalies in long videos, we propose a novel glance and focus network to effectively integrate spatial-temporal information for accurate anomaly detection. In addition, we empirically found that existing approaches that use feature magnitudes to represent the degree of anomalies typically ignore the effects of scene variations, and hence result in sub-optimal performance due to the inconsistency of feature magnitudes across scenes. To a…
KAUST（沙特阿拉伯阿卜杜拉国王科技大学）；University of Amsterdam（阿姆斯特丹大学）；Qualcomm Technologies Inc（高通技术公司）
This paper addresses the problem of spatiotemporal localization of actions in videos. Compared to leading approaches, which all learn to localize based on carefully annotated boxes on training video frames, we adhere to a weakly-supervised solution that only requires a video class label. We introduce an actor-supervised architecture that exploits the inherent compositionality of actions in terms of actor transformations, to localize actions. We make two contributions. First, we propose actor proposals derived from a detector for human and non-human actors intended for images, which is linked o…
We present a new framework for Patch Distribution Modeling, PaDiM, to concurrently detect and localize anomalies in images in a one-class learning setting. PaDiM makes use of a pretrained convolutional neural network (CNN) for patch embedding, and of multivariate Gaussian distributions to get a probabilistic representation of the normal class. It also exploits correlations between the different semantic levels of CNN to better localize anomalies. PaDiM outperforms current state-of-the-art approaches for both anomaly detection and localization on the MVTec AD and STC datasets. To match real-wor…
Nanjing University of Science and Technology（南京理工大学）
Video anomaly detection under video-level labels is currently a challenging task. Previous works have made progresses on discriminating whether a video sequencecontains anomalies. However, most of them fail to accurately localize the anomalous events within videos in the temporal domain. In this paper, we propose a Weakly Supervised Anomaly Localization (WSAL) method focusing on temporally localizing anomalous segments within anomalous videos. Inspired by the appearance difference in anomalous videos, the evolution of adjacent temporal segments is evaluated for the localization of anomalous se…
University of Michigan（密歇根大学）；Nvidia Corporation（英伟达公司）
This paper proposes a new image-based localization framework that explicitly localizes the camera/robot by fusing Convolutional Neural Network (CNN) and sequential images' geometric constraints. The camera is localized using a single or few observed images and training images with 6-degree-of-freedom pose labels. A Siamese network structure is adopted to train an image descriptor network, and the visually similar candidate image in the training set is retrieved to localize the testing image geometrically. Meanwhile, a probabilistic motion model predicts the pose based on a constant velocity as…
Videos can be manipulated by duplicating a sequence of consecutive frames with the goal of concealing or imitating a specific content in the same video. In this paper, we propose a novel coarse-to-fine framework based on deep Convolutional Neural Networks to automatically detect and localize such frame duplication. First, an I3D network finds coarse-level matches between candidate duplicated frame sequences and the corresponding selected original frame sequences. Then a Siamese network based on ResNet architecture identifies fine-level correspondences between an individual duplicated frame and…
Malek Ashtar University of Technology（马利克·阿什塔尔理工大学）；Iran University of Science and Technology（伊朗科技大学）；Auckland University of Technology（奥克兰理工大学）
In this paper, we propose a method for real-time anomaly detection and localization in crowded scenes. Each video is defined as a set of non-overlapping cubic patches, and is described using two local and global descriptors. These descriptors capture the video properties from different aspects. By incorporating simple and cost-effective Gaussian classifiers, we can distinguish normal activities and anomalies in videos. The local and global features are based on structure similarity between adjacent patches and the features learned in an unsupervised way, using a sparse auto- encoder. Experimen…
Information Technology University（信息技术大学）；University of Central Florida（中佛罗里达大学）
Surveillance videos are able to capture a variety of realistic anomalies. In this paper, we propose to learn anomalies by exploiting both normal and anomalous videos. To avoid annotating the anomalous segments or clips in training videos, which is very time consuming, we propose to learn anomaly through the deep multiple instance ranking framework by leveraging weakly labeled training videos, i.e. the training labels (anomalous or normal) are at video-level instead of clip-level. In our approach, we consider normal and anomalous videos as bags and video segments as instances in multiple instan…
Topo-R1: Detecting Topological Anomalies via Vision-Language Models
Stony Brook University（石溪大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）；Stanford University（斯坦福大学）；Penn State University（宾夕法尼亚州立大学）
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Edinburgh（爱丁堡大学）；Khalifa University（哈利法大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
Boosting Fine-Grained Visual Anomaly Detection with Coarse-Knowledge-Aware Adversarial Learning
Sun Yat-sen University（中山大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；The Hong Kong Polytechnic University（香港理工大学）；Pazhou Lab（琶洲实验室）
Unsupervised Patch-GAN with Targeted Patch Ranking for Fine-Grained Novelty Detection in Medical Imaging
the Southern University of Science and Technology（南方科技大学）；University of Oxford（牛津大学）；Tsinghua University（清华大学）；University of Warwick（华威大学）；Northwest University（西北大学）；Peng Cheng Lab（鹏城实验室）
