Incipient anomalies present milder symptoms compared to severe ones, and are more difficult to detect and diagnose due to their close resemblance to normal operating conditions. The lack of incipient anomaly examples in the training data can pose severe risks to anomaly detection methods that are built upon Machine Learning (ML) techniques, because these anomalies can be easily mistaken as normal operating conditions. To address this challenge, we propose to utilize the uncertainty information available from ensemble learning to identify potential misclassified incipient anomalies. We show in this paper that ensemble learning methods can give improved performance on incipient anomalies and identify common pitfalls in these models through extensive experiments on two real-world datasets. Then, we discuss how to design more effective ensemble models for detecting incipient anomalies.
University of California, Berkeley（加州大学伯克利分校）；Arizona State University（亚利桑那州立大学）
Incipient anomalies present milder symptoms compared to severe ones, and are more difficult to detect and diagnose due to their close resemblance to normal operating conditions. The lack of incipient anomaly examples in the training data can pose severe risks to anomaly detection methods that are built upon Machine Learning (ML) techniques, because these anomalies can be easily mistaken as normal operating conditions. To address this challenge, we propose to utilize the uncertainty information available from ensemble learning to identify potential misclassified incipient anomalies. We show in …
NTT Secure Platform Laboratories（NTT安全平台实验室）；NTT Communication Science Laboratories（NTT通信科学实验室）；NTT Software Innovation Center（NTT软件创新中心）
We propose the Autoencoding Binary Classifiers (ABC), a novel supervised anomaly detector based on the Autoencoder (AE). There are two main approaches in anomaly detection: supervised and unsupervised. The supervised approach accurately detects the known anomalies included in training data, but it cannot detect the unknown anomalies. Meanwhile, the unsupervised approach can detect both known and unknown anomalies that are located away from normal data points. However, it does not detect known anomalies as accurately as the supervised approach. Furthermore, even if we have labeled normal data p…
Detecting Anomalies in Blockchain Transactions using Machine Learning Classifiers and Explainability Analysis
Premier University（总理大学）；United International University（联合国际大学）；Cyber Security Cooperative Research Centre（网络安全合作研究中心）；Edith Cowan University（埃迪斯科文大学）
Bridging Unsupervised and Semi-Supervised Anomaly Detection: A Theoretically-Grounded and Practical Framework with Synthetic Anomalies
School of Cybersecurity and Privacy, Georgia Institute of Technology（网络安全与隐私学院，佐治亚理工学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（H. Milton Stewart工业与系统工程学院，佐治亚理工学院）
University of Bologna（博洛尼亚大学）；INFN（意大利国家核物理研究所）；Federal University of Lavras（拉夫拉斯联邦大学）；INFN-CNAF（意大利国家核物理研究所国家核物理计算中心）
Log-based predictive maintenance of computing centers is a main concern regarding the worldwide computing grid that supports the CERN (European Organization for Nuclear Research) physics experiments. A log, as event-oriented adhoc information, is quite often given as unstructured big data. Log data processing is a time-consuming computational task. The goal is to grab essential information from a continuously changeable grid environment to construct a classification model. Evolving granular classifiers are suited to learn from time-varying log streams and, therefore, perform online classificat…
A Multivariate Statistical Framework for Detection, Classification and Pre-localization of Anomalies in Water Distribution Networks
Institute of Computer Science and Information Technologies, National Technical University "Kharkiv Polytechnic Institute"（计算机科学与信息科技研究所，国家技术大学"Kharkiv 工程学院"）；Informatics Institute, University of Amsterdam（信息学院，阿姆斯特丹大学）；Bernoulli Institute, University of Groningen（伯努利研究所，格罗宁根大学）
A Classifier-Based Approach to Multi-Class Anomaly Detection for Astronomical Transients
Irvington High School（欧文顿高中）；University of the Western Cape（西开普大学）；South African Radio Astronomy Observatory（南非射电天文观测台）；Massachusetts Institute of Technology（麻省理工学院）
This paper proposes an autoencoder (AE) that is used for improving the performance of once-class classifiers for the purpose of detecting anomalies. Traditional one-class classifiers (OCCs) perform poorly under certain conditions such as high-dimensionality and sparsity. Also, the size of the training set plays an important role on the performance of one-class classifiers. Autoencoders have been widely used for obtaining useful latent variables from high-dimensional datasets. In the proposed approach, the AE is capable of deriving meaningful features from high-dimensional datasets while doing …
Anomaly detection is a fundamental yet challenging problem in machine learning due to the lack of label information. In this work, we propose a novel and powerful framework, dubbed as SLA$^2$P, for unsupervised anomaly detection. After extracting representative embeddings from raw data, we apply random projections to the features and regard features transformed by different projections as belonging to distinct pseudo classes. We then train a classifier network on these transformed features to perform self-supervised learning. Next we add adversarial perturbation to the transformed features to …
More than a thousand 8" silicon sensors will be visually inspected to look for anomalies on their surface during the quality control preceding assembly into the High-Granularity Calorimeter for the CMS experiment at CERN. A deep learning-based algorithm that pre-selects potentially anomalous images of the sensor surface in real time has been developed to automate the visual inspection. The anomaly detection is done by an ensemble of independent deep convolutional neural networks: an autoencoder and a classifier. The performance is evaluated on images acquired in production. The pre-selection r…
Centre of Excellence for Advanced Sensor Technology (CEASTech), Universiti Malaysia Perlis（马来西亚玻璃市大学先进传感器技术卓越中心）
In this study, an early fire detection algorithm has been proposed based on low cost array sensing system, utilizing gas sensors, dust particles and ambient sensors such as temperature and humidity sensor. The odor or smell-print emanated from various fire sources and building construction materials at early stage are measured. For this purpose, odor profile data from five common fire sources and three common building construction materials were used to develop the classification model. Normalized feature extractions of the smell print data were performed before subjected to prediction classif…
Center for Applied Intelligent Systems Research (CAISR)（应用智能系统研究中心）；Halmstad University（哈尔姆斯塔德大学）
In contextual anomaly detection, an object is only considered anomalous within a specific context. Most existing methods for CAD use a single context based on a set of user-specified contextual features. However, identifying the right context can be very challenging in practice, especially in datasets, with a large number of attributes. Furthermore, in real-world systems, there might be multiple anomalies that occur in different contexts and, therefore, require a combination of several "useful" contexts to unveil them. In this work, we leverage active learning and ensembles to effectively dete…
A Classifier-Based Approach to Multi-Class Anomaly Detection Applied to Astronomical Time-Series
Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology（麻省理工学院卡弗里天体物理与空间研究所）；Irvington High School（欧文顿高中）；Department of Physics and Astronomy, University of the Western Cape（西开普大学物理与天文学系）；South African Radio Astronomy Observatory（南非射电天文观测台）
Learning to Detect Cyber Attacks: Neural Anomaly Detection for Cybersecurity with Theoretical Insights
Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）；School of Cybersecurity and Privacy, Georgia Institute of Technology（佐治亚理工学院网络安全与隐私学院）；H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院H·米尔顿·斯图尔特工业与系统工程学院）
Semi-supervised anomaly detection is a common problem, as often the datasets containing anomalies are partially labeled. We propose a canonical framework: Semi-supervised Pseudo-labeler Anomaly Detection with Ensembling (SPADE) that isn't limited by the assumption that labeled and unlabeled data come from the same distribution. Indeed, the assumption is often violated in many applications - for example, the labeled data may contain only anomalies unlike unlabeled data, or unlabeled data may contain different types of anomalies, or labeled data may contain only 'easy-to-label' samples. SPADE ut…
Extreme pricing anomalies may occur unexpectedly without a trivial cause, and equity traders typically experience a meticulous process to source disparate information and analyze its reliability before integrating it into the trusted knowledge base. We introduce DeepTrust, a reliable financial knowledge retrieval framework on Twitter to explain extreme price moves at speed, while ensuring data veracity using state-of-the-art NLP techniques. Our proposed framework consists of three modules, specialized for anomaly detection, information retrieval and reliability assessment. The workflow starts …
The number of end devices that use the last mile wireless connectivity is dramatically increasing with the rise of smart infrastructures and require reliable functioning to support smooth and efficient business processes. To efficiently manage such massive wireless networks, more advanced and accurate network monitoring and malfunction detection solutions are required. In this paper, we perform a first time analysis of image-based representation techniques for wireless anomaly detection using recurrence plots and Gramian angular fields and propose a new deep learning architecture enabling accu…
University of Cambridge（剑桥大学）；Massachusetts Institute of Technology（麻省理工学院）；The Alan Turing Institute（艾伦·图灵研究所）；University of the Western Cape（西开普大学）；South African Radio Astronomy Observatory（南非射电天文台）；African Institute for Mathematical Sciences（非洲数学科学研究所）；Swinburne University of Technology（斯威本科技大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
New time-domain surveys, such as the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), will observe millions of transient alerts each night, making standard approaches of visually identifying new and interesting transients infeasible. We present two novel methods of automatically detecting anomalous transient light curves in real-time. Both methods are based on the simple idea that if the light curves from a known population of transients can be accurately modelled, any deviations from model predictions are likely anomalies. The first modelling approach is a probabilistic neura…
Jožef Stefan Institute（约瑟夫·斯特凡研究所）；Jožef Stefan International Postgraduate School（约瑟夫·斯特凡国际研究生院）；Bitlis Eren University（比特利斯·埃伦大学）；University of Ljubljana（卢布尔雅那大学）
After decades of research, the Internet of Things (IoT) is finally permeating real-life and helps improve the efficiency of infrastructures and processes as well as our health. As a massive number of IoT devices are deployed, they naturally incur great operational costs to ensure intended operations. To effectively handle such intended operations in massive IoT networks, automatic detection of malfunctioning, namely anomaly detection, becomes a critical but challenging task. In this paper, motivated by a real-world experimental IoT deployment, we introduce four types of wireless network anomal…
With the wider availability of sensor technology, a number of Structural Health Monitoring (SHM) systems are deployed to monitor civil infrastructure. The continuous monitoring provides valuable information about the structure that can help in providing a decision support system for retrofits and other structural modifications. However, when the sensors are exposed to harsh environmental conditions, the data measured by the SHM systems tend to be affected by multiple anomalies caused by faulty or broken sensors. Given a deluge of high-dimensional data collected continuously over time, research…
Technische Universität Dresden（德累斯顿工业大学）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；University of Padova（帕多瓦大学）；M31
Smartphones and wearable devices are fast growing technologies that, in conjunction with advances in wireless sensor hardware, are enabling ubiquitous sensing applications. Wearables are suitable for indoor and outdoor scenarios, can be placed on many parts of the human body and can integrate a large number of sensors capable of gathering physiological and behavioral biometric information. Here, we are concerned with gait analysis systems that extract meaningful information from a user's movements to identify anomalies and changes in their walking style. The solution that is put forward is sub…
Warsaw University of Technology（华沙理工大学）；TECNALIA（泰克纳利亚）；University of the Basque Country (UPV/EHU)（巴斯克大学(UPV/EHU)）
Unsupervised anomaly discovery in stream data is a research topic with many practical applications. However, in many cases, it is not easy to collect enough training data with labeled anomalies for supervised learning of an anomaly detector in order to deploy it later for identification of real anomalies in streaming data. It is thus important to design anomalies detectors that can correctly detect anomalies without access to labeled training data. Our idea is to adapt the Online evolving Spiking Neural Network (OeSNN) classifier to the anomaly detection task. As a result, we offer an Online e…
Columbia University（哥伦比亚大学）；Facebook Data Engineering（Facebook数据工程）
Methods for unsupervised anomaly detection suffer from the fact that the data is unlabeled, making it difficult to assess the optimality of detection algorithms. Ensemble learning has shown exceptional results in classification and clustering problems, but has not seen as much research in the context of outlier detection. Existing methods focus on combining output scores of individual detectors, but this leads to outputs that are not easily interpretable. In this paper, we introduce a theoretical foundation for combining individual detectors with Bayesian classifier combination. Not only are p…
Technische Universität Berlin（柏林工业大学）；Memorial Sloan-Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；University of Göttingen（哥廷根大学）；Technische Universität Darmstadt（达姆施塔特工业大学）；German Institute for International Educational Research（德国国际教育研究所）
Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to gr…
Delft University of Technology（代尔夫特理工大学）；Irdeto B.V.（爱迪德公司）；Eindhoven University of Technology（埃因霍温理工大学）
This paper focuses on detecting anomalies in a digital video broadcasting (DVB) system from providers' perspective. We learn a probabilistic deterministic real timed automaton profiling benign behavior of encryption control in the DVB control access system. This profile is used as a one-class classifier. Anomalous items in a testing sequence are detected when the sequence is not accepted by the learned model.
We proposed a multivariate time series anomaly detection frame-work Ymir, which leverages ensemble learning and supervisedlearning technology to efficiently learn and adapt to anomaliesin real-world system applications. Ymir integrates several currentlywidely used unsupervised anomaly detection models through anensemble learning method, and thus can provide robust frontalanomaly detection results in unsupervised scenarios. In a super-vised setting, domain experts and system users discuss and providelabels (anomalous or not) for the training data, which reflects theiranomaly detection criteria …
Unravelling physics beyond the standard model with classical and quantum anomaly detection
IBM Research（IBM研究院）；ETH Zürich（苏黎世联邦理工学院）；Autonomous University of Barcelona（巴塞罗那自治大学）；European Organization for Nuclear Research (CERN)（欧洲核子研究组织（CERN））；PASQAL SAS（PASQAL SAS公司）
Anomaly detection (AD), separating anomalies from normal data, has many applications across domains, from security to healthcare. While most previous works were shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-superv…
AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning
European Space Agency (ESA)（欧洲航天局）；European Space Astronomy Centre (ESAC)（欧洲空间天文学中心）；Astronomisches Rechen-Institut（天文计算研究所）；Zentrum für Astronomie der Universität Heidelberg（海德堡大学天文研究中心）；Kapteyn Astronomical Institute（Kapteyn天文研究所）；University of Groningen（格罗宁根大学）
Wroclaw University of Science and Technology（弗罗茨瓦夫理工大学）
Hypertext transfer protocol (HTTP) is one of the most widely used protocols on the Internet. As a consequence, most attacks (i.e., SQL injection, XSS) use HTTP as the transport mechanism. Therefore, it is crucial to develop an intelligent solution that would allow to effectively detect and filter out anomalies in HTTP traffic. Currently, most of the anomaly detection systems are either rule-based or trained using manually selected features. We propose utilizing modern unsupervised language representation model for embedding HTTP requests and then using it to classify anomalies in the traffic. …
Support vector data description (SVDD) is a popular technique for detecting anomalies. The SVDD classifier partitions the whole space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD classifier requires a kernel function, and the Gaussian kernel is a common choice for the kernel function. The Gaussian kernel has a bandwidth parameter, whose value is important for good results. A small bandwidth leads to overfitting, and the resulting SVDD classifier overestimates …
Department of Computer Science, Princess Sumaya University for Technology（科技大学公主苏玛亚计算机科学系）
One of the most effective threats that targeting cybercriminals to limit network performance is Denial of Service (DOS) attack. Thus, data security, completeness and efficiency could be greatly damaged by this type of attacks. This paper developed a network traffic system that relies on adopted dataset to differentiate the DOS attacks from normal traffic. The detection model is built with five Rule-based machine learning classifiers (DecisionTable, JRip, OneR, PART and ZeroR). The findings have shown that the ICMP variables are implemented in the identification of ICMP attack, HTTP flood attac…
Data is one of the essential ingredients to power deep learning research. Small datasets, especially specific to medical institutes, bring challenges to deep learning training stage. This work aims to develop a practical deep multimodal that can classify patients into abnormal and normal categories accurately as well as assist radiologists to detect visual and textual anomalies by locating areas of interest. The detection of the anomalies is achieved through a novel technique which extends the integrated gradients methodology with an unsupervised clustering algorithm. This technique also intro…
University of Michigan（密歇根大学）；West Virginia University（西弗吉尼亚大学）
In this paper, we propose a general framework to learn a robust large-margin binary classifier when corrupt measurements, called anomalies, caused by sensor failure might be present in the training set. The goal is to minimize the generalization error of the classifier on non-corrupted measurements while controlling the false alarm rate associated with anomalous samples. By incorporating a non-parametric regularizer based on an empirical entropy estimator, we propose a Geometric-Entropy-Minimization regularized Maximum Entropy Discrimination (GEM-MED) method to learn to classify and detect ano…
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）
With the recent developments in artificial intelligence and machine learning, anomalies in network traffic can be detected using machine learning approaches. Before the rise of machine learning, network anomalies which could imply an attack, were detected using well-crafted rules. An attacker who has knowledge in the field of cyber-defence could make educated guesses to sometimes accurately predict which particular features of network traffic data the cyber-defence mechanism is looking at. With this information, the attacker can circumvent a rule-based cyber-defense system. However, after the …
University of Michigan（密歇根大学）；West Virginia University（西弗吉尼亚大学）
In this paper, we propose a general framework to learn a robust large-margin binary classifier when corrupt measurements, called anomalies, caused by sensor failure might be present in the training set. The goal is to minimize the generalization error of the classifier on non-corrupted measurements while controlling the false alarm rate associated with anomalous samples. By incorporating a non-parametric regularizer based on an empirical entropy estimator, we propose a Geometric-Entropy-Minimization regularized Maximum Entropy Discrimination (GEM-MED) method to learn to classify and detect ano…
Classifier-Free Diffusion-Based Weakly-Supervised Approach for Health Indicator Derivation in Rotating Machines: Advancing Early Fault Detection and Condition Monitoring
Columbia University in the City of New York（纽约市哥伦比亚大学）；University of Florida（佛罗里达大学）
As engineered systems grow in complexity, there is an increasing need for automatic methods that can detect, diagnose, and even correct transient anomalies that inevitably arise and can be difficult or impossible to diagnose and fix manually. Among the most sensitive and complex systems of our civilization are the detectors that search for incredibly small variations in distance caused by gravitational waves -- phenomena originally predicted by Albert Einstein to emerge and propagate through the universe as the result of collisions between black holes and other massive objects in deep space. T…
