We present a meta-learning framework for weakly supervised anomaly detection in videos, where the detector learns to adapt to unseen types of abnormal activities effectively when only video-level annotations of binary labels are available. Our work is motivated by the fact that existing methods suffer from poor generalization to diverse unseen examples. We claim that an anomaly detector equipped with a meta-learning scheme alleviates the limitation by leading the model to an initialization point for better optimization. We evaluate the performance of our framework on two challenging datasets, UCF-Crime and ShanghaiTech. The experimental results demonstrate that our algorithm boosts the capability to localize unseen abnormal events in a weakly supervised setting. Besides the technical contributions, we perform the annotation of missing labels in the UCF-Crime dataset and make our task evaluated effectively.
A classical approach to abnormal activity detection is to learn a representation for normal activities from the training data and then use this learned representation to detect abnormal activities while testing. Typically, the methods based on this approach operate at a fixed timescale - either a single time-instant (eg. frame-based) or a constant time duration (eg. video-clip based). But human abnormal activities can take place at different timescales. For example, jumping is a short term anomaly and loitering is a long term anomaly in a surveillance scenario. A single and pre-defined timesca…
We present a method to learn a joint multimodal representation space that enables recognition of unseen activities in videos. We first compare the effect of placing various constraints on the embedding space using paired text and video data. We also propose a method to improve the joint embedding space using an adversarial formulation, allowing it to benefit from unpaired text and video data. By using unpaired text data, we show the ability to learn a representation that better captures unseen activities. In addition to testing on publicly available datasets, we introduce a new, large-scale te…
Google Health, Google（谷歌健康，谷歌）；Apollo Radiology International（阿波罗国际放射医学）；Northwestern Medicine（西北医学）
Chest radiography (CXR) is the most widely-used thoracic clinical imaging modality and is crucial for guiding the management of cardiothoracic conditions. The detection of specific CXR findings has been the main focus of several artificial intelligence (AI) systems. However, the wide range of possible CXR abnormalities makes it impractical to build specific systems to detect every possible condition. In this work, we developed and evaluated an AI system to classify CXRs as normal or abnormal. For development, we used a de-identified dataset of 248,445 patients from a multi-city hospital networ…
National Taiwan University（台湾大学）；Inventec Corporation（英业达股份有限公司）
Anomaly detection aims to identify abnormal data that deviates from the normal ones, while typically requiring a sufficient amount of normal data to train the model for performing this task. Despite the success of recent anomaly detection methods, performing anomaly detection in an unseen domain remain a challenging task. In this paper, we address the task of domain-generalized textured surface anomaly detection. By observing normal and abnormal surface data across multiple source domains, our model is expected to be generalized to an unseen textured surface of interest, in which only a small …
Singapore Management University（新加坡管理大学）；Zhejiang University（浙江大学）；The University of Adelaide（阿德莱德大学）
Despite most existing anomaly detection studies assume the availability of normal training samples only, a few labeled anomaly examples are often available in many real-world applications, such as defect samples identified during random quality inspection, lesion images confirmed by radiologists in daily medical screening, etc. These anomaly examples provide valuable knowledge about the application-specific abnormality, enabling significantly improved detection of similar anomalies in some recent models. However, those anomalies seen during training often do not illustrate every possible class…
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）
Action localization in untrimmed videos is an important topic in the field of video understanding. However, existing action localization methods are restricted to a pre-defined set of actions and cannot localize unseen activities. Thus, we consider a new task to localize unseen activities in videos via image queries, named Image-Based Activity Localization. This task faces three inherent challenges: (1) how to eliminate the influence of semantically inessential contents in image queries; (2) how to deal with the fuzzy localization of inaccurate image queries; (3) how to determine the precise b…
Xi’an Jiaotong University（西安交通大学）；Monash University（莫纳什大学）；The University of Queensland（昆士兰大学）；Carnegie Mellon University（卡内基梅隆大学）
An integral part of video analysis and surveillance is temporal activity detection, which means to simultaneously recognize and localize activities in long untrimmed videos. Currently, the most effective methods of temporal activity detection are based on deep learning, and they typically perform very well with large scale annotated videos for training. However, these methods are limited in real applications due to the unavailable videos about certain activity classes and the time-consuming data annotation. To solve this challenging problem, we propose a novel task setting called zero-shot tem…
Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection
School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Harvard University（哈佛大学）
University of Twente（特文特大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）；Zhejiang University（浙江大学）
In this paper, we present an unsupervised learning framework for analyzing activities and interactions in surveillance videos. In our framework, three levels of video events are connected by Hierarchical Dirichlet Process (HDP) model: low-level visual features, simple atomic activities, and multi-agent interactions. Atomic activities are represented as distribution of low-level features, while complicated interactions are represented as distribution of atomic activities. This learning process is unsupervised. Given a training video sequence, low-level visual features are extracted based on opt…
Abnormal activity recognition requires detection of occurrence of anomalous events that suffer from a severe imbalance in data. In a video, normal is used to describe activities that conform to usual events while the irregular events which do not conform to the normal are referred to as abnormal. It is far more common to observe normal data than to obtain abnormal data in visual surveillance. In this paper, we propose an approach where we can obtain abnormal data by transforming normal data. This is a challenging task that is solved through a multi-stage pipeline approach. We utilize a number …
UniVAD v2: Unified Visual Anomaly Detection via Support-Conditioned Boundary Construction
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Yinwang Intelligent Technology Co. Ltd.（银河智能科技有限公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Wuhan AI Research（武汉人工智能研究所）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Intellectual Property & Big Data（广东省知识产权与大数据重点实验室）
University of California, Santa Barbara（加州大学圣巴巴拉分校）；The Ohio State University（俄亥俄州立大学）
Although promising results have been achieved in video captioning, existing models are limited to the fixed inventory of activities in the training corpus, and do not generalize to open vocabulary scenarios. Here we introduce a novel task, zero-shot video captioning, that aims at describing out-of-domain videos of unseen activities. Videos of different activities usually require different captioning strategies in many aspects, i.e. word selection, semantic construction, and style expression etc, which poses a great challenge to depict novel activities without paired training data. But meanwhil…
Pázmány Péter Catholic University（帕兹马尼·彼得天主教大学）；Autonomous University of Madrid（马德里自治大学）；University of Bordeaux（波尔多大学）；CNRS（法国国家科学研究中心）；Univ. Bordeaux（波尔多大学）；Bordeaux INP（波尔多国立理工学院）；LaBRI（波尔多计算机科学研究实验室）
Advances in deep-learning-based pipelines have led to breakthroughs in a variety of microscopy image diagnostics. However, a sufficiently big training data set is usually difficult to obtain due to high annotation costs. In the case of banded chromosome images, the creation of big enough libraries is difficult for multiple pathologies due to the rarity of certain genetic disorders. Generative Adversarial Networks (GANs) have proven to be effective in generating synthetic images and extending training data sets. In our work, we implement a conditional adversarial network that allows generation …
DictAS: A Framework for Class-Generalizable Few-Shot Anomaly Segmentation via Dictionary Lookup
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Casivision；Longmen Laboratory（龙门实验室）；HDU；UTS；UCLA（加州大学洛杉矶分校）；Tsinghua University（清华大学）
In recent years, video conferencing (VC) popularity has skyrocketed for a wide range of activities. As a result, the number of VC users surged sharply. The sharp increase in VC usage has been accompanied by various newly emerging privacy and security challenges. VC meetings became a target for various security attacks, such as Zoombombing. Other VC-related challenges also emerged. For example, during COVID lockdowns, educators had to teach in online environments struggling with keeping students engaged for extended periods. In parallel, the amount of available VC videos has grown exponentially…
Malaviya National Institute of Technology Jaipur（斋浦尔马拉维亚国家技术学院）
Time-stamp aware anomaly detection in traffic videos is an essential task for the advancement of the intelligent transportation system. Anomaly detection in videos is a challenging problem due to sparse occurrence of anomalous events, inconsistent behavior of a different type of anomalies and imbalanced available data for normal and abnormal scenarios. In this paper, we present a three-stage pipeline to learn the motion patterns in videos to detect a visual anomaly. First, the background is estimated from recent history frames to identify the motionless objects. This background image is used t…
KITE – Toronto Rehabilitation Institute, University Health Network（KITE – 多伦多康复研究所，大学健康网络）；University of Toronto（多伦多大学）
Fall detection is an important problem from both the health and machine learning perspective. A fall can lead to severe injuries, long term impairments or even death in some cases. In terms of machine learning, it presents a severely class imbalance problem with very few or no training data for falls owing to the fact that falls occur rarely. In this paper, we take an alternate philosophy to detect falls in the absence of their training data, by training the classifier on only the normal activities (that are available in abundance) and identifying a fall as an anomaly. To realize such a classi…
Anomaly detection is a classical problem in computer vision, namely the determination of the normal from the abnormal when datasets are highly biased towards one class (normal) due to the insufficient sample size of the other class (abnormal). While this can be addressed as a supervised learning problem, a significantly more challenging problem is that of detecting the unknown/unseen anomaly case that takes us instead into the space of a one-class, semi-supervised learning paradigm. We introduce such a novel anomaly detection model, by using a conditional generative adversarial network that jo…
Video anomaly detection is a core problem in vision. Correctly detecting and identifying anomalous behaviors in pedestrians from video data will enable safety-critical applications such as surveillance, activity monitoring, and human-robot interaction. In this paper, we propose to leverage trajectory localization and prediction for unsupervised pedestrian anomaly event detection. Different than previous reconstruction-based approaches, our proposed framework rely on the prediction errors of normal and abnormal pedestrian trajectories to detect anomalies spatially and temporally. We present exp…
Polytechnique Montréal（蒙特利尔理工学院）；Institute of Biomedical Engineering（生物医学工程研究所）
We present a local anomaly detection method in videos. As opposed to most existing methods that are computationally expensive and are not very generalizable across different video scenes, we propose an adversarial framework that learns the temporal local appearance variations by predicting the appearance of a normally behaving object in the next frame of a scene by only relying on its current and past appearances. In the presence of an abnormally behaving object, the reconstruction error between the real and the predicted next appearance of that object indicates the likelihood of an anomaly. O…
Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information（视觉信息工程国家级工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Robotics, Xi'an Jiaotong University（机器人学，西安交通大学）；School of Electronics and Control Engineering（电子控制工程学院）
FaceTracer: Unveiling Source Identities from Swapped Face Images and Videos for Fraud Prevention
School of Cyber Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（科技研究局前沿人工智能研究中心）；College of Computing and Data Science at Nanyang Technological University（南洋理工大学计算与数据科学学院）
Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
Future human action forecasting from partial observations of activities is an important problem in many practical applications such as assistive robotics, video surveillance and security. We present a method to forecast actions for the unseen future of the video using a neural machine translation technique that uses encoder-decoder architecture. The input to this model is the observed RGB video, and the objective is to forecast the correct future symbolic action sequence. Unlike prior methods that make action predictions for some unseen percentage of video one for each frame, we predict the co…
Self-supervised Pseudo Multi-class Pre-training for Unsupervised Anomaly Detection and Segmentation in Medical Images
Harvard Medical School（哈佛医学院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）；Singapore Management University（新加坡管理大学）；University of Surrey（萨里大学）
Columbia University（哥伦比亚大学）；University of Connecticut（康涅狄格大学）；NEC Laboratories America, Inc.（NEC美国实验室）
Detecting abnormal activities in real-world surveillance videos is an important yet challenging task as the prior knowledge about video anomalies is usually limited or unavailable. Despite that many approaches have been developed to resolve this problem, few of them can capture the normal spatio-temporal patterns effectively and efficiently. Moreover, existing works seldom explicitly consider the local consistency at frame level and global coherence of temporal dynamics in video sequences. To this end, we propose Convolutional Transformer based Dual Discriminator Generative Adversarial Network…
BenchReAD: A systematic benchmark for retinal anomaly detection
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China（香港理工大学护理学院智能健康中心）；School of Biomedical Engineering, Tsinghua University, Beijing, China（清华大学生物医学工程学院）；Research Center for Medical AI, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（中国科学院深圳先进技术研究院医学人工智能研究中心）
