Recently, people tried to use a few anomalies for video anomaly detection (VAD) instead of only normal data during the training process. A side effect of data imbalance occurs when a few abnormal data face a vast number of normal data. The latest VAD works use triplet loss or data re-sampling strategy to lessen this problem. However, there is still no elaborately designed structure for discriminative VAD with a few anomalies. In this paper, we propose a DiscRiminative-gEnerative duAl Memory (DREAM) anomaly detection model to take advantage of a few anomalies and solve data imbalance. We use two shallow discriminators to tighten the normal feature distribution boundary along with a generator for the next frame prediction. Further, we propose a dual memory module to obtain a sparse feature representation in both normality and abnormality space. As a result, DREAM not only solves the data imbalance problem but also learn a reasonable feature space. Further theoretical analysis shows that our DREAM also works for the unknown anomalies. Comparing with the previous methods on UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech, our model outperforms all the baselines with no extra parameters. The ablation study demonstrates the effectiveness of our dual memory module and discriminative-generative network.
Anomaly detection is an important problem in computer vision; however, the scarcity of anomalous samples makes this task difficult. Thus, recent anomaly detection methods have used only normal images with no abnormal areas for training. In this work, a powerful anomaly detection method is proposed based on student-teacher feature pyramid matching (STPM), which consists of a student and teacher network. Generative models are another approach to anomaly detection. They reconstruct normal images from an input and compute the difference between the predicted normal and the input. Unfortunately, ST…
Jiangxi University of Finance and Economics（江西财经大学）
Anomaly detection in surveillance videos is a challenging task due to the diversity of anomalous video content and duration. In this paper, we consider video anomaly detection as a regression problem with respect to anomaly scores of video clips under weak supervision. Hence, we propose an anomaly detection framework, called Anomaly Regression Net (AR-Net), which only requires video-level labels in training stage. Further, to learn discriminative features for anomaly detection, we design a dynamic multiple-instance learning loss and a center loss for the proposed AR-Net. The former is used to …
University of Ljubljana（卢布尔雅那大学）；Faculty of Computer Information Science（计算机信息科学学院）
Visual surface anomaly detection aims to detect local image regions that significantly deviate from normal appearance. Recent surface anomaly detection methods rely on generative models to accurately reconstruct the normal areas and to fail on anomalies. These methods are trained only on anomaly-free images, and often require hand-crafted post-processing steps to localize the anomalies, which prohibits optimizing the feature extraction for maximal detection capability. In addition to reconstructive approach, we cast surface anomaly detection primarily as a discriminative problem and propose a …
Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；Institute of Robotics and Intelligent Manufacturing, Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）机器人与智能制造研究院）
As a kind of generative self-supervised learning methods, generative adversarial nets have been widely studied in the field of anomaly detection. However, the representation learning ability of the generator is limited since it pays too much attention to pixel-level details, and generator is difficult to learn abstract semantic representations from label prediction pretext tasks as effective as discriminator. In order to improve the representation learning ability of generator, we propose a self-supervised learning framework combining generative methods and discriminative methods. The generato…
We introduce a powerful student-teacher framework for the challenging problem of unsupervised anomaly detection and pixel-precise anomaly segmentation in high-resolution images. Student networks are trained to regress the output of a descriptive teacher network that was pretrained on a large dataset of patches from natural images. This circumvents the need for prior data annotation. Anomalies are detected when the outputs of the student networks differ from that of the teacher network. This happens when they fail to generalize outside the manifold of anomaly-free training data. The intrinsic u…
Recent efforts towards video anomaly detection (VAD) try to learn a deep autoencoder to describe normal event patterns with small reconstruction errors. The video inputs with large reconstruction errors are regarded as anomalies at the test time. However, these methods sometimes reconstruct abnormal inputs well because of the powerful generalization ability of deep autoencoder. To address this problem, we present a novel approach for anomaly detection, which utilizes discriminative prototypes of normal data to reconstruct video frames. In this way, the model will favor the reconstruction of no…
Video anomaly detection is recently formulated as a multiple instance learning task under weak supervision, in which each video is treated as a bag of snippets to be determined whether contains anomalies. Previous efforts mainly focus on the discrimination of the snippet itself without modeling the temporal dynamics, which refers to the variation of adjacent snippets. Therefore, we propose a Discriminative Dynamics Learning (DDL) method with two objective functions, i.e., dynamics ranking loss and dynamics alignment loss. The former aims to enlarge the score dynamics gap between positive and n…
NVIDIA Research（英伟达研究院）；NVIDIA CAI（英伟达人工智能计算中心）；University of Technology Sydney（悉尼科技大学）；Australian National University（澳大利亚国立大学）
Person re-identification (re-id) remains challenging due to significant intra-class variations across different cameras. Recently, there has been a growing interest in using generative models to augment training data and enhance the invariance to input changes. The generative pipelines in existing methods, however, stay relatively separate from the discriminative re-id learning stages. Accordingly, re-id models are often trained in a straightforward manner on the generated data. In this paper, we seek to improve learned re-id embeddings by better leveraging the generated data. To this end, we …
We address an anomaly detection setting in which training sequences are unavailable and anomalies are scored independently of temporal ordering. Current algorithms in anomaly detection are based on the classical density estimation approach of learning high-dimensional models and finding low-probability events. These algorithms are sensitive to the order in which anomalies appear and require either training data or early context assumptions that do not hold for longer, more complex videos. By defining anomalies as examples that can be distinguished from other examples in the same video, our def…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We present a new "learning-to-learn"-type approach that enables rapid learning of concepts from small-to-medium sized training sets and is primarily designed for web-initialized image retrieval. At the core of our approach is a deep architecture (a Set2Model network) that maps sets of examples to simple generative probabilistic models such as Gaussians or mixtures of Gaussians in the space of high-dimensional descriptors. The parameters of the embedding into the descriptor space are trained in the end-to-end fashion in the meta-learning stage using a set of training learning problems. The main…
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）
One pivot challenge for image anomaly (AD) detection is to learn discriminative information only from normal class training images. Most image reconstruction based AD methods rely on the discriminative capability of reconstruction error. This is heuristic as image reconstruction is unsupervised without incorporating normal-class-specific information. In this paper, we propose an AD method called dual deep reconstruction networks based image decomposition (DDR-ID). The networks are trained by jointly optimizing for three losses: the one-class loss, the latent space constrain loss and the recons…
Weakly supervised video anomaly detection (WSVAD) is a challenging task since only video-level labels are available for training. In previous studies, the discriminative power of the learned features is not strong enough, and the data imbalance resulting from the mini-batch training strategy is ignored. To address these two issues, we propose a novel WSVAD method based on cross-batch clustering guidance. To enhance the discriminative power of features, we propose a batch clustering based loss to encourage a clustering branch to generate distinct normal and abnormal clusters based on a batch of…
The University of Sydney（悉尼大学）；Capital Markets CRC（资本市场合作研究中心）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Unlike conventional anomaly detection research that focuses on point anomalies, our goal is to detect anomalous collections of individual data points. In particular, we perform group anomaly detection (GAD) with an emphasis on irregular group distributions (e.g. irregular mixtures of image pixels). GAD is an important task in detecting unusual and anomalous phenomena in real-world applications such as high energy particle physics, social media, and medical imaging. In this paper, we take a generative approach by proposing deep generative models: Adversarial autoencoder (AAE) and variational au…
Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
University of Saskatchewan（萨斯喀彻温大学）；Saskatchewan Health Authority（萨斯喀彻温卫生局）
Wireless Capsule Endoscopy (WCE) helps physicians examine the gastrointestinal (GI) tract noninvasively. There are few studies that address pathological assessment of endoscopy images in multiclass classification and most of them are based on binary anomaly detection or aim to detect a specific type of anomaly. Multiclass anomaly detection is challenging, especially when the dataset is poorly sampled or imbalanced. Many available datasets in endoscopy field, such as KID2, suffer from an imbalance issue, which makes it difficult to train a high-performance model. Additionally, increasing the nu…
Masked Face Recognition with Generative-to-Discriminative Representations
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security at University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Cloud Music Inc.（网易云音乐）；Department of Communication Engineering, Shanghai University（上海大学通信工程系）
Singapore Management University（新加坡管理大学）；Institute for Infocomm Research（资讯通信研究院）；Australia National University（澳大利亚国立大学）
Most classifiers rely on discriminative boundaries that separate instances of each class from everything else. We argue that discriminative boundaries are counter-intuitive as they define semantics by what-they-are-not; and should be replaced by generative classifiers which define semantics by what-they-are. Unfortunately, generative classifiers are significantly less accurate. This may be caused by the tendency of generative models to focus on easy to model semantic generative factors and ignore non-semantic factors that are important but difficult to model. We propose a new generative model …
SARD: Segmentation-Aware Anomaly Synthesis via Region-Constrained Diffusion with Discriminative Mask Guidance
Global Institute of Future Technology（未来技术全球研究所）；Shanghai Jiao Tong University（上海交通大学）；Department of Intelligent Manufacturing（智能制造系）；Contemporary Amperex Technology Co.,Ltd（当代电子技术有限公司）
Georgia State University（佐治亚州立大学）；Etsy Inc.（Etsy公司）
Diffusion Denoising Probability Models (DDPM) and Vision Transformer (ViT) have demonstrated significant progress in generative tasks and discriminative tasks, respectively, and thus far these models have largely been developed in their own domains. In this paper, we establish a direct connection between DDPM and ViT by integrating the ViT architecture into DDPM, and introduce a new generative model called Generative ViT (GenViT). The modeling flexibility of ViT enables us to further extend GenViT to hybrid discriminative-generative modeling, and introduce a Hybrid ViT (HybViT). Our work is am…
Australian National University, Canberra（堪培拉澳大利亚国立大学）
One-class learning is the classic problem of fitting a model to data for which annotations are available only for a single class. In this paper, we propose a novel objective for one-class learning. Our key idea is to use a pair of orthonormal frames -- as subspaces -- to "sandwich" the labeled data via optimizing for two objectives jointly: i) minimize the distance between the origins of the two subspaces, and ii) to maximize the margin between the hyperplanes and the data, either subspace demanding the data to be in its positive and negative orthant respectively. Our proposed objective howeve…
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Center on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；Samsung AI Center（三星人工智能中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Target domain pseudo-labelling has shown effectiveness in unsupervised domain adaptation (UDA). However, pseudo-labels of unlabeled target domain data are inevitably noisy due to the distribution shift between source and target domains. This paper proposes a Generative model-based Noise-Robust Training method (GeNRT), which eliminates domain shift while mitigating label noise. GeNRT incorporates a Distribution-based Class-wise Feature Augmentation (D-CFA) and a Generative-Discriminative classifier Consistency (GDC), both based on the class-wise target distributions modelled by generative model…
Zhejiang University（浙江大学）；University of Technology Sydney（悉尼科技大学）；Baidu Research（百度研究院）
Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanw…
SynDoc: A Hybrid Discriminative-Generative Framework for Enhancing Synthetic Domain-Adaptive Document Key Information Extraction
The University of Western Australia（西澳大学）；The University of Melbourne（墨尔本大学）；The University of Sydney（悉尼大学）；Weill Cornell Medicine, Cornell University（韦尔·科恩医学中心，康奈尔大学）
IR-Flow: Bridging Discriminative and Generative Image Restoration via Rectified Flow
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, School of Information Science and Technology, University of Science and Technology of China（脑启发智能感知与认知联合实验室，信息科学与技术学院，中国科学技术大学）
University of Pittsburgh（匹兹堡大学）；University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）
We propose an attention-based method that aggregates local image features to a subject-level representation for predicting disease severity. In contrast to classical deep learning that requires a fixed dimensional input, our method operates on a set of image patches; hence it can accommodate variable length input image without image resizing. The model learns a clinically interpretable subject-level representation that is reflective of the disease severity. Our model consists of three mutually dependent modules which regulate each other: (1) a discriminative network that learns a fixed-length …
We study unsupervised learning by developing introspective generative modeling (IGM) that attains a generator using progressively learned deep convolutional neural networks. The generator is itself a discriminator, capable of introspection: being able to self-evaluate the difference between its generated samples and the given training data. When followed by repeated discriminative learning, desirable properties of modern discriminative classifiers are directly inherited by the generator. IGM learns a cascade of CNN classifiers using a synthesis-by-classification algorithm. In the experiments, …
University College London（伦敦大学学院）；University of Cambridge（剑桥大学）
The task of predicting human motion is complicated by the natural heterogeneity and compositionality of actions, necessitating robustness to distributional shifts as far as out-of-distribution (OoD). Here we formulate a new OoD benchmark based on the Human3.6M and CMU motion capture datasets, and introduce a hybrid framework for hardening discriminative architectures to OoD failure by augmenting them with a generative model. When applied to current state-of-the-art discriminative models, we show that the proposed approach improves OoD robustness without sacrificing in-distribution performance,…
Deep generative models (DGMs) are effective on learning multilayered representations of complex data and performing inference of input data by exploring the generative ability. However, little work has been done on examining or empowering the discriminative ability of DGMs on making accurate predictions. This paper presents max-margin deep generative models (mmDGMs), which explore the strongly discriminative principle of max-margin learning to improve the discriminative power of DGMs, while retaining the generative capability. We develop an efficient doubly stochastic subgradient algorithm for…
The pattern theory of Grenander is a mathematical framework where patterns are represented by probability models on random variables of algebraic structures. In this paper, we review three families of probability models, namely, the discriminative models, the descriptive models, and the generative models. A discriminative model is in the form of a classifier. It specifies the conditional probability of the class label given the input signal. A descriptive model specifies the probability distribution of the signal, based on an energy function defined on the signal. A generative model assumes th…
Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification…
Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey
University of California Los Angeles（加利福尼亚大学洛杉矶分校）；Rice University（莱斯大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Walmart Global Tech（沃尔玛全球技术公司）；Santa Clara University（圣克拉拉大学）；Shanghai AI Laboratory（上海人工智能实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Research Cambridge（微软研究院剑桥实验室）
Computer vision is hard because of a large variability in lighting, shape, and texture; in addition the image signal is non-additive due to occlusion. Generative models promised to account for this variability by accurately modelling the image formation process as a function of latent variables with prior beliefs. Bayesian posterior inference could then, in principle, explain the observation. While intuitively appealing, generative models for computer vision have largely failed to deliver on that promise due to the difficulty of posterior inference. As a result the community has favoured effic…
