Anomaly Detection and Localization based on Double Kernelized Scoring and Matrix Kernels
Deloitte Touche Tohmatsu LLC（德勤东京有限公司）
Shunsuke Hirose, Tomotake Kozu, Yingzi Jin
备注：
15 pages and 3 figures. A translation of "Anomaly Detection based on Doubly-Kernelized Scoring and Matrix Kernels" by Shunsuke Hirose and Tomotake Kozu (This article is written in Japanese). DOI: https://doi.org/10.1527/tjsai.AI30-D
Anomaly detection is necessary for proper and safe operation of large-scale systems consisting of multiple devices, networks, and/or plants. Those systems are often characterized by a pair of multivariate datasets. To detect anomaly in such a system and localize element(s) associated with anomaly, one would need to estimate scores that quantify anomalousness of the entire system as well as its elements. However, it is not trivial to estimate such scores by considering changes of relationships between the elements, which strongly correlate with each other. Moreover, it is necessary to estimate the scores for the entire system and its elements from a single framework, in order to identify relationships among the scores for localizing elements associated with anomaly. Here, we developed a new method to quantify anomalousness of an entire system and its elements simultaneously.
The purpose of this paper is threefold. The first one is to propose a new anomaly detection method: Double Kernelized Scoring (DKS). DKS is a unified framework for entire-system anomaly scoring and element-wise anomaly scoring. Therefore, DKS allows for conducting simultaneously 1) anomaly detection for the entire system and 2) localization for identifying faulty elements responsible for the system anomaly. The second purpose is to propose a new kernel function: Matrix Kernel. The Matrix Kernel is defined between general matrices, which might have different dimensions, allowing for conducting anomaly detection on systems where the number of elements change over time. The third purpose is to demonstrate the effectiveness of the proposed method experimentally. We evaluated the proposed method with synthetic and real time series data. The results demonstrate that DKS is able to detect anomaly and localize the elements associated with it successfully.
School of Computing, National University of Singapore（新加坡国立大学计算学院）；Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）
How can we detect anomalies: that is, samples that significantly differ from a given set of high-dimensional data, such as images or sensor data? This is a practical problem with numerous applications and is also relevant to the goal of making learning algorithms more robust to unexpected inputs. Autoencoders are a popular approach, partly due to their simplicity and their ability to perform dimension reduction. However, the anomaly scoring function is not adaptive to the natural variation in reconstruction error across the range of normal samples, which hinders their ability to detect real an…
Nowadays, many classification algorithms have been applied to various industries to help them work out their problems met in real-life scenarios. However, in many binary classification tasks, samples in the minority class only make up a small part of all instances, which leads to the datasets we get usually suffer from high imbalance ratio. Existing models sometimes treat minority classes as noise or ignore them as outliers encountering data skewing. In order to solve this problem, we propose a bagging ensemble learning framework $ASE$ (Anomaly Scoring Based Ensemble Learning). This framework …
Anomaly detection aims to identify observations that deviate from the typical pattern of data. Anomalous observations may correspond to financial fraud, health risks, or incorrectly measured data in practice. We show detecting anomalies in high-dimensional mixed data is enhanced through first embedding the data then assessing an anomaly scoring scheme. We focus on unsupervised detection and the continuous and categorical (mixed) variable case. We propose a kurtosis-weighted Factor Analysis of Mixed Data for anomaly detection, FAMDAD, to obtain a continuous embedding for anomaly scoring. We ill…
Nowadays, advanced intrusion detection systems (IDSs) rely on a combination of anomaly detection and signature-based methods. An IDS gathers observations, analyzes behavioral patterns, and reports suspicious events for further investigation. A notorious issue anomaly detection systems (ADSs) and IDSs face is the possibility of high false alarms, which even state-of-the-art systems have not overcome. This is especially a problem with large and complex systems. The number of non-critical alarms can easily overwhelm administrators and increase the likelihood of ignoring future alerts. Mitigation …
Anomaly detection becomes increasingly important for the dependability and serviceability of IT services. As log lines record events during the execution of IT services, they are a primary source for diagnostics. Thereby, unsupervised methods provide a significant benefit since not all anomalies can be known at training time. Existing unsupervised methods need anomaly examples to obtain a suitable decision boundary required for the anomaly detection task. This requirement poses practical limitations. Therefore, we develop A2Log, which is an unsupervised anomaly detection method consisting of t…
Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This paper presents a systematic and comprehensive evaluation of unsupervised and semi-supervised deep-learning based methods for anomaly detection and diagnosis on multivariate time series data from cyberphysical systems. Unlike previous works, we vary the model and post-processing of model errors, i.e. the scoring functions independently of each other, through a grid of 10 models and 4 scoring functions, comparing these…
We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical $M$-estimation. We interpret the KDE based on a radial, positive semi-definite kernel as a sample mean in the associated reproducing kernel Hilbert space. Since the sample mean is sensitive to outliers, we estimate it robustly via $M$-estimation, yielding a robust kernel density estimator (RKDE). An RKDE can be computed efficiently via a kernelized iterativel…
Arizona State University（亚利桑那州立大学）；Georgia Institute of Technology（佐治亚理工学院）
In point-based sensing systems such as coordinate measuring machines (CMM) and laser ultrasonics where complete sensing is impractical due to the high sensing time and cost, adaptive sensing through a systematic exploration is vital for online inspection and anomaly quantification. Most of the existing sequential sampling methodologies focus on reducing the overall fitting error for the entire sampling space. However, in many anomaly quantification applications, the main goal is to estimate sparse anomalous regions in the pixel-level accurately. In this paper, we develop a novel framework name…
Practical anomaly detection requires applying numerous approaches due to the inherent difficulty of unsupervised learning. Direct comparison between complex or opaque anomaly detection algorithms is intractable; we instead propose a framework for associating the scores of multiple methods. Our aim is to answer the question: how should one measure the similarity between anomaly scores generated by different methods? The scoring crux is the extremes, which identify the most anomalous observations. A pair of algorithms are defined here to be similar if they assign their highest scores to roughly …
Network troubleshooting is still a heavily human-intensive process. To reduce the time spent by human operators in the diagnosis process, we present a system based on (i) unsupervised learning methods for detecting anomalies in the time domain, (ii) an attention mechanism to rank features in the feature space and finally (iii) an expert knowledge module able to seamlessly incorporate previously collected domain-knowledge. In this paper, we thoroughly evaluate the performance of the full system and of its individual building blocks: particularly, we consider (i) 10 anomaly detection algorithms …
Anomaly detection presents a unique challenge in machine learning, due to the scarcity of labeled anomaly data. Recent work attempts to mitigate such problems by augmenting training of deep anomaly detection models with additional labeled anomaly samples. However, the labeled data often does not align with the target distribution and introduces harmful bias to the trained model. In this paper, we aim to understand the effect of a biased anomaly set on anomaly detection. Concretely, we view anomaly detection as a supervised learning task where the objective is to optimize the recall at a given …
Budapest University of Technology and Economics（布达佩斯技术与经济大学）；University of Debrecen（德布勒森大学）
The anomaly detection method presented by this paper has a special feature: it does not only indicate whether an observation is anomalous or not but also tells what exactly makes an anomalous observation unusual. Hence, it provides support to localize the reason of the anomaly. The proposed approach is model-based; it relies on the multivariate probability distribution associated with the observations. Since the rare events are present in the tails of the probability distributions, we use copula functions, that are able to model the fat-tailed distributions well. The presented procedure scales…
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；The University of Adelaide（阿德莱德大学）
Although deep learning has been applied to successfully address many data mining problems, relatively limited work has been done on deep learning for anomaly detection. Existing deep anomaly detection methods, which focus on learning new feature representations to enable downstream anomaly detection methods, perform indirect optimization of anomaly scores, leading to data-inefficient learning and suboptimal anomaly scoring. Also, they are typically designed as unsupervised learning due to the lack of large-scale labeled anomaly data. As a result, they are difficult to leverage prior knowledge …
Isolation-based Spherical Ensemble Representations for Tabular Anomaly Detection
Tsinghua Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院，清华大学，中国）；School of Computing and Information Technology, Great Bay University, China（计算机与信息科技学院，Great Bay大学，中国）；School of Compuer Science, Nanjing University, China（计算机科学学院，南京大学，中国）；Computer department, Dongguan university of technology, China（计算机系，东莞科技学院，中国）；College of Computer Science and Technology, China University of Petroleum, China（计算机科学与技术学院，中国石油大学，中国）；School of IT, Deakin University, Australia（信息技术学院，德金大学，澳大利亚）
NTT Communication Science Laboratories（NTT通信科学实验室）
We propose a method for maximizing a partial area under a receiver operating characteristic (ROC) curve (pAUC) for binary classification tasks. In binary classification tasks, accuracy is the most commonly used as a measure of classifier performance. In some applications such as anomaly detection and diagnostic testing, accuracy is not an appropriate measure since prior probabilties are often greatly biased. Although in such cases the pAUC has been utilized as a performance measure, few methods have been proposed for directly maximizing the pAUC. This optimization is achieved by using a scorin…
Anomaly detection methods strive to discover patterns that differ from the norm in a semantic way. This goal is ambiguous as a data point differing from the norm by an attribute e.g., age, race or gender, may be considered anomalous by some operators while others may consider this attribute irrelevant. Breaking from previous research, we present a new anomaly detection method that allows operators to exclude an attribute from being considered as relevant for anomaly detection. Our approach then learns representations which do not contain information over the nuisance attributes. Anomaly scorin…
Checking various log files from different processes can be a tedious task as these logs contain lots of events, each with a (possibly large) number of attributes. We developed a way to automatically model log files and detect outlier traces in the data. For that we extend Dynamic Bayesian Networks to model the normal behavior found in log files. We introduce a new algorithm that is able to learn a model of a log file starting from the data itself. The model is capable of scoring traces even when new values or new combinations of values appear in the log file.
NUS（新加坡国立大学）；Institute for Infocomm Research (IR) A*STAR（新加坡科技研究局资讯通信研究院）
Anomaly detection in multi-variate time series (MVTS) data is a huge challenge as it requires simultaneous representation of long term temporal dependencies and correlations across multiple variables. More often, this is solved by breaking the complexity through modeling one dependency at a time. In this paper, we propose a Time-series Representational Learning through Contrastive Predictive Coding (TRL-CPC) towards anomaly detection in MVTS data. First, we jointly optimize an encoder, an auto-regressor and a non-linear transformation function to effectively learn the representations of the MV…
University at Buffalo, State University of New York (SUNY)（纽约州立大学布法罗分校）
Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with "known" number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable def…
Anomaly Detection via Mean Shift Density Enhancement
School of Data Science, Indian Institute of Science Education and Research（数据科学学院，印度科学教育与研究学院）；Institute of Computer Science, University of Rostock（计算机科学研究所，罗斯托克大学）；Leibniz-Institute for Food Systems Biology, Technical University of Munich（食品系统生物学莱比锡研究所，慕尼黑技术大学）；Stellenbosch Institute of Advanced Studies (STIAS)（斯托尔波茨堡高级研究 institute (STIAS)）
Deep Probabilistic Modeling of User Behavior for Anomaly Detection via Mixture Density Networks
University of California, Berkeley（加州大学伯克利分校）；University of Southern California（南加州大学）；University of Washington（华盛顿大学）；Northeastern University（东北大学）；Northwest Missouri State University（西北密苏里州立大学）；Georgia Institute of Technology（佐治亚理工学院）
Graph Neural AI with Temporal Dynamics for Comprehensive Anomaly Detection in Microservices
Boston University（波士顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Hofstra University（霍夫斯特拉大学）；University of Southern California（南加州大学）
University of Victoria（维多利亚大学）；Helwan University（赫尔万大学）
The visualization and detection of anomalies (outliers) are of crucial importance to many fields, particularly cybersecurity. Several approaches have been proposed in these fields, yet to the best of our knowledge, none of them has fulfilled both objectives, simultaneously or cooperatively, in one coherent framework. The visualization methods of these approaches were introduced for explaining the output of a detection algorithm, not for data exploration that facilitates a standalone visual detection. This is our point of departure: UN-AVOIDS, an unsupervised and nonparametric approach for both…
We introduce a new unsupervised anomaly detection ensemble called SPI which can harness privileged information - data available only for training examples but not for (future) test examples. Our ideas build on the Learning Using Privileged Information (LUPI) paradigm pioneered by Vapnik et al. [19,17], which we extend to unsupervised learning and in particular to anomaly detection. SPI (for Spotting anomalies with Privileged Information) constructs a number of frames/fragments of knowledge (i.e., density estimates) in the privileged space and transfers them to the anomaly scoring space through…
Entropy Causal Graphs for Multivariate Time Series Anomaly Detection
Institute of Innovation, Science and Sustainability（创新、科学与可持续性研究所）；Federation University Australia, Australia and CSIRO's Data61（联邦大学澳大利亚分校和CSIRO的数据61）；Federation University Australia（联邦大学澳大利亚分校）；RMIT University（皇家墨尔本理工学院）
Much of the world's data is streaming, time-series data, where anomalies give significant information in critical situations; examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. There are no benchmarks to adequately test and score the efficacy of real-time anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which attempts to provide a controlled and repeatable environment o…
