Anomaly detection is a method for discovering unusual and suspicious behavior. In many real-world scenarios, the examined events can be directly linked to the actions of an adversary, such as attacks on computer networks or frauds in financial operations. While the defender wants to discover such malicious behavior, the attacker seeks to accomplish their goal (e.g., exfiltrating data) while avoiding the detection. To this end, anomaly detectors have been used in a game-theoretic framework that captures these goals of a two-player competition. We extend the existing models to more realistic settings by (1) allowing both players to have continuous action spaces and by assuming that (2) the defender cannot perfectly observe the action of the attacker. We propose two algorithms for solving such games -- a direct extension of existing algorithms based on discretizing the feature space and linear programming and the second algorithm based on constrained learning. Experiments show that both algorithms are applicable for cases with low feature space dimensions but the learning-based method produces less exploitable strategies and it is scalable to higher dimensions. Moreover, we use real-world data to compare our approaches with existing classifiers in a data-exfiltration scenario via the DNS channel. The results show that our models are significantly less exploitable by an informed attacker.
ETH Zurich（苏黎世联邦理工学院）；California Institute of Technology（加州理工学院）；CERN（欧洲核子研究组织）
We apply an Adversarially Learned Anomaly Detection (ALAD) algorithm to the problem of detecting new physics processes in proton-proton collisions at the Large Hadron Collider. Anomaly detection based on ALAD matches performances reached by Variational Autoencoders, with a substantial improvement in some cases. Training the ALAD algorithm on 4.4 fb-1 of 8 TeV CMS Open Data, we show how a data-driven anomaly detection and characterization would work in real life, re-discovering the top quark by identifying the main features of the t-tbar experimental signature at the LHC.
Institute of Industrial Science（产业技术研究所）；The University of Tokyo（东京大学）
Detection of emerging topics are now receiving renewed interest motivated by the rapid growth of social networks. Conventional term-frequency-based approaches may not be appropriate in this context, because the information exchanged are not only texts but also images, URLs, and videos. We focus on the social aspects of theses networks. That is, the links between users that are generated dynamically intentionally or unintentionally through replies, mentions, and retweets. We propose a probability model of the mentioning behaviour of a social network user, and propose to detect the emergence of …
AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning
European Space Agency (ESA)（欧洲航天局）；European Space Astronomy Centre (ESAC)（欧洲空间天文学中心）；Astronomisches Rechen-Institut（天文计算研究所）；Zentrum für Astronomie der Universität Heidelberg（海德堡大学天文研究中心）；Kapteyn Astronomical Institute（Kapteyn天文研究所）；University of Groningen（格罗宁根大学）
Anomaly detectors are often used to produce a ranked list of statistical anomalies, which are examined by human analysts in order to extract the actual anomalies of interest. Unfortunately, in realworld applications, this process can be exceedingly difficult for the analyst since a large fraction of high-ranking anomalies are false positives and not interesting from the application perspective. In this paper, we aim to make the analyst's job easier by allowing for analyst feedback during the investigation process. Ideally, the feedback influences the ranking of the anomaly detector in a way th…
Washington State University（华盛顿州立大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Human analysts that use anomaly detection systems in practice want to retain the use of simple and explainable global anomaly detectors. In this paper, we propose a novel human-in-the-loop learning algorithm called GLAD (GLocalized Anomaly Detection) that supports global anomaly detectors. GLAD automatically learns their local relevance to specific data instances using label feedback from human analysts. The key idea is to place a uniform prior on the relevance of each member of the anomaly detection ensemble over the input feature space via a neural network trained on unlabeled instances. Sub…
In critical applications of anomaly detection including computer security and fraud prevention, the anomaly detector must be configurable by the analyst to minimize the effort on false positives. One important way to configure the anomaly detector is by providing true labels for a few instances. We study the problem of label-efficient active learning to automatically tune anomaly detection ensembles and make four main contributions. First, we present an important insight into how anomaly detector ensembles are naturally suited for active learning. This insight allows us to relate the greedy qu…
Traditional anomaly detection on social media mostly focuses on individual point anomalies while anomalous phenomena usually occur in groups. Therefore it is valuable to study the collective behavior of individuals and detect group anomalies. Existing group anomaly detection approaches rely on the assumption that the groups are known, which can hardly be true in real world social media applications. In this paper, we take a generative approach by proposing a hierarchical Bayes model: Group Latent Anomaly Detection (GLAD) model. GLAD takes both pair-wise and point-wise data as input, automatica…
Modern distributed cyber-physical systems (CPSs) encounter a large variety of physical faults and cyber anomalies and in many cases, they are vulnerable to catastrophic fault propagation scenarios due to strong connectivity among the sub-systems. This paper presents a new data-driven framework for system-wide anomaly detection for addressing such issues. The framework is based on a spatiotemporal feature extraction scheme built on the concept of symbolic dynamics for discovering and representing causal interactions among the subsystems of a CPS. The extracted spatiotemporal features are then u…
Ulsan National Institute of Science and Technology (UNIST)（蔚山国家科学技术研究所）
While several techniques for detecting trace-level anomalies in event logs in offline settings have appeared recently in the literature, such techniques are currently lacking for online settings. Event log anomaly detection in online settings can be crucial for discovering anomalies in process execution as soon as they occur and, consequently, allowing to promptly take early corrective actions. This paper describes a novel approach to event log anomaly detection on event streams that uses statistical leverage. Leverage has been used extensively in statistics to develop measures to identify out…
While analyzing vehicular sensor data, we found that frequently occurring waveforms could serve as features for further analysis, such as rule mining, classification, and anomaly detection. The discovery of waveform patterns, also known as time-series motifs, has been studied extensively; however, available techniques for discovering frequently occurring time-series motifs were found lacking in either efficiency or quality: Standard subsequence clustering results in poor quality, to the extent that it has even been termed 'meaningless'. Variants of hierarchical clustering using techniques for …
Columbia University（哥伦比亚大学）；Princeton University（普林斯顿大学）；Flatiron Institute（弗莱迪龙研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Supernovae mark the explosive deaths of stars and enrich the cosmos with heavy elements. Future telescopes will discover thousands of new supernovae nightly, creating a need to flag astrophysically interesting events rapidly for followup study. Ideally, such an anomaly detection pipeline would be independent of our current knowledge and be sensitive to unexpected phenomena. Here we present an unsupervised method to search for anomalous time series in real time for transient, multivariate, and aperiodic signals. We use a RNN-based variational autoencoder to encode supernova time series and an i…
Two Is Better Than One: Aligned Representation Pairs for Anomaly Detection
Department of Computer Science ETH Zurich（计算机科学系，苏黎世联邦理工学院）；Research Center Trustworthy Data Science and Security of the University Alliance Ruhr（可信数据科学与安全的鲁尔大学联盟研究中心）；Department of Statistics TU Dortmund University（统计学系，多特蒙德技术大学）
Event datasets in the financial domain are often constructed based on actual application scenarios, and their event types are weakly reusable due to scenario constraints; at the same time, the massive and diverse new financial big data cannot be limited to the event types defined for specific scenarios. This limitation of a small number of event types does not meet our research needs for more complex tasks such as the prediction of major financial events and the analysis of the ripple effects of financial events. In this paper, a three-stage approach is proposed to accomplish incremental disco…
Yonsei University（延世大学）；Wizcore Co., Ltd.（Wizcore有限公司）；Department of Computational Science and Engineering, Yonsei University（延世大学计算科学与工程系）；Department of Electrical and Electronic Engineering, Yonsei University（延世大学电气与电子工程系）；Institute of Information and Communications Technology Planning Evaluation（信息通信技术规划评估研究所）；Yonsei University Research Fund（延世大学研究基金）
We introduce a data-driven anomaly detection framework using a manufacturing dataset collected from a factory assembly line. Given heterogeneous time series data consisting of operation cycle signals and sensor signals, we aim at discovering abnormal events. Motivated by our empirical findings that conventional single-stage benchmark approaches may not exhibit satisfactory performance under our challenging circumstances, we propose a two-stage deep anomaly detection (TDAD) framework in which two different unsupervised learning models are adopted depending on types of signals. In Stage I, we se…
Performance monitoring, anomaly detection, and root-cause analysis in complex cyber-physical systems (CPSs) are often highly intractable due to widely diverse operational modes, disparate data types, and complex fault propagation mechanisms. This paper presents a new data-driven framework for root-cause analysis, based on a spatiotemporal graphical modeling approach built on the concept of symbolic dynamics for discovering and representing causal interactions among sub-systems of complex CPSs. We formulate the root-cause analysis problem as a minimization problem via the proposed inference bas…
National Sun Yat-sen University（国立中山大学）；University of Pittsburgh（匹兹堡大学）
Process Monitoring involves tracking a system's behaviors, evaluating the current state of the system, and discovering interesting events that require immediate actions. In this paper, we consider monitoring temporal system state sequences to help detect the changes of dynamic systems, check the divergence of the system development, and evaluate the significance of the deviation. We begin with discussions of data reduction, symbolic data representation, and the anomaly detection in temporal discrete sequences. Time-series representation methods are also discussed and used in this paper to disc…
Discovering Antagonists in Networks of Systems: Robot Deployment
Institute of Engineering and Computational Mechanics, University of Stuttgart（工程与计算力学研究所，斯图加特大学）；Department of Mechanical Engineering, LUT University（机械工程系，卢托大学）
We describe two recently proposed machine learning approaches for discovering emerging trends in fatal accidental drug overdoses. The Gaussian Process Subset Scan enables early detection of emerging patterns in spatio-temporal data, accounting for both the non-iid nature of the data and the fact that detecting subtle patterns requires integration of information across multiple spatial areas and multiple time steps. We apply this approach to 17 years of county-aggregated data for monthly opioid overdose deaths in the New York City metropolitan area, showing clear advantages in the utility of di…
