This paper presents a novel application of explainable AI (XAI) for root-causing performance degradation in machine learning models that learn continuously from user engagement data. In such systems a single feature corruption can cause cascading feature, label and concept drifts. We have successfully applied this technique to improve the reliability of models used in personalized advertising. Performance degradation in such systems manifest as prediction anomalies in the models. These models are typically trained continuously using features that are produced by hundreds of real time data processing pipelines or derived from other upstream models. A failure in any of these pipelines or an instability in any of the upstream models can cause feature corruption, causing the model's predicted output to deviate from the actual output and the training data to become corrupted. The causal relationship between the features and the predicted output is complex, and root-causing is challenging due to the scale and dynamism of the system. We demonstrate how temporal shifts in the global feature importance distribution can effectively isolate the cause of a prediction anomaly, with better recall than model-to-feature correlation methods. The technique appears to be effective even when approximating the local feature importance using a simple perturbation-based method, and aggregating over a few thousand examples. We have found this technique to be a model-agnostic, cheap and effective way to monitor complex data pipelines in production and have deployed a system for continuously analyzing the global feature importance distribution of continuously trained models.
Deep learning has now become the de facto approach to the recognition of anomalies in medical imaging. Their 'black box' way of classifying medical images into anomaly labels poses problems for their acceptance, particularly with clinicians. Current explainable AI methods offer justifications through visualizations such as heat maps but cannot guarantee that the network is focusing on the relevant image region fully containing the anomaly. In this paper, we develop an approach to explainable AI in which the anomaly is assured to be overlapping the expected location when present. This is made p…
Google（谷歌）；The George Washington University（乔治华盛顿大学）
The need for explainable AI (XAI) is well established but relatively little has been published outside of the supervised learning paradigm. This paper focuses on a principled approach to applying explainability and interpretability to the task of unsupervised anomaly detection. We argue that explainability is principally an algorithmic task and interpretability is principally a cognitive task, and draw on insights from the cognitive sciences to propose a general-purpose method for practical diagnosis using explained anomalies. We define Attribution Error, and demonstrate, using real-world labe…
Telefónica（西班牙电信公司）；Universidad Politécnica de Madrid（马德里理工大学）
Identifying anomalies in the fuel consumption of the vehicles of a fleet is a crucial aspect for optimizing consumption and reduce costs. However, this information alone is insufficient, since fleet operators need to know the causes behind anomalous fuel consumption. We combine unsupervised anomaly detection techniques, domain knowledge and interpretable Machine Learning models for explaining potential causes of abnormal fuel consumption in terms of feature relevance. The explanations are used for generating recommendations about fuel optimization, that are adjusted according to two different …
Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Temporal Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis
Complying with the EU AI Act: Innovations in Explainable and User-Centric Hand Gesture Recognition
Technical University Munich（慕尼黑工业大学）；Infineon Technologies AG（英飞凌科技股份公司）；Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校）；Software Competence Center Hagenberg GmbH (SCCH)（哈根堡软件能力中心有限公司（SCCH））
National University of Singapore（新加坡国立大学）；Trustwave（Trustwave（特鲁斯特威）公司）
This paper looks into the problem of detecting network anomalies by analyzing NetFlow records. While many previous works have used statistical models and machine learning techniques in a supervised way, such solutions have the limitations that they require large amount of labeled data for training and are unlikely to detect zero-day attacks. Existing anomaly detection solutions also do not provide an easy way to explain or identify attacks in the anomalous traffic. To address these limitations, we develop and present GEE, a framework for detecting and explaining anomalies in network traffic. G…
Towards Differentiating Between Failures and Domain Shifts in Industrial Data Streams
Jagiellonian Human-Centered AI Lab, Mark Kac Center for Complex Systems Research, Institute of Applied Computer Science, Faculty of Physics, Astronomy and Applied Computer Science, Jagiellonian University（雅盖隆大学人中心人工智能实验室、马克·卡中心复杂系统研究所以及应用计算机科学研究所、物理、天文学与应用计算机科学学院）；Institute of Computing Science, Poznań University of Technology（波兹南技术大学计算科学研究所）
Existing anomaly detection paradigms overwhelmingly focus on training detection models using exclusively normal data or unlabeled data (mostly normal samples). One notorious issue with these approaches is that they are weak in discriminating anomalies from normal samples due to the lack of the knowledge about the anomalies. Here, we study the problem of few-shot anomaly detection, in which we aim at using a few labeled anomaly examples to train sample-efficient discriminative detection models. To address this problem, we introduce a novel weakly-supervised anomaly detection framework to train …
LUT University（拉彭兰塔-拉赫蒂工业大学）；Centre Tecnològic de Telecomunicacions de Catalunya (CTTC/CERCA)（加泰罗尼亚电信技术中心（CTTC/CERCA））；Aalborg University（奥尔堡大学）；The American College of Greece（希腊美国学院）
Timely and accurate detection of anomalies in power electronics is becoming increasingly critical for maintaining complex production systems. Robust and explainable strategies help decrease system downtime and preempt or mitigate infrastructure cyberattacks. This work begins by explaining the types of uncertainty present in current datasets and machine learning algorithm outputs. Three techniques for combating these uncertainties are then introduced and analyzed. We further present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer,…
The paper proposes summarized attribution-based post-hoc explanations for the detection and identification of bias in data. A global explanation is proposed, and a step-by-step framework on how to detect and test bias is introduced. Since removing unwanted bias is often a complicated and tremendous task, it is automatically inserted, instead. Then, the bias is evaluated with the proposed counterfactual approach. The obtained results are validated on a sample skin lesion dataset. Using the proposed method, a number of possible bias causing artifacts are successfully identified and confirmed in …
The Chinese University of Hong Kong（香港中文大学）；University of Minnesota（明尼苏达大学）
In explainable artificial intelligence, discriminative feature localization is critical to reveal a blackbox model's decision-making process from raw data to prediction. In this article, we use two real datasets, the MNIST handwritten digits and MIT-BIH Electrocardiogram (ECG) signals, to motivate key characteristics of discriminative features, namely adaptiveness, predictive importance and effectiveness. Then, we develop a localization framework based on adversarial attacks to effectively localize discriminative features. In contrast to existing heuristic methods, we also provide a statistica…
Cybersecurity threat detection based on a UEBA framework using Deep Autoencoders
Galician Research and Development Center in Advanced Telecommunications (Gradiant)（加利西亚先进电信研究与发展中心（Gradiant））；Universidade de Vigo（维戈大学）；Galician Centre for Mathematical Research and Technology (CITMAga)（加利西亚数学研究与技术中心（CITMAga））
MINES: Explainable Anomaly Detection through Web API Invariant Inference
National University of Singapore（国立新加坡大学）；Shanghai Jiao Tong University（上海交通大学）；Singapore Management University（新加坡管理学院）；GovTech Singapore（新加坡政府科技局）；Chongqing University（重庆大学）
University of Arizona（亚利桑那大学）；Rowan University（罗文大学）
Unexplainable black-box models create scenarios where anomalies cause deleterious responses, thus creating unacceptable risks. These risks have motivated the field of eXplainable Artificial Intelligence (XAI) to improve trust by evaluating local interpretability in black-box neural networks. Unfortunately, the ground truth is unavailable for the model's decision, so evaluation is limited to qualitative assessment. Further, interpretability may lead to inaccurate conclusions about the model or a false sense of trust. We propose to improve XAI from the vantage point of the user's trust by explor…
Anomaly detection is critical for finding suspicious behavior in innumerable systems. We need to detect anomalies in real-time, i.e. determine if an incoming entity is anomalous or not, as soon as we receive it, to minimize the effects of malicious activities and start recovery as soon as possible. Therefore, online algorithms that can detect anomalies in a streaming manner are essential. We first propose MIDAS which uses a count-min sketch to detect anomalous edges in dynamic graphs in an online manner, using constant time and memory. We then propose two variants, MIDAS-R which incorporates t…
Deutsche Bundesbank（德意志联邦银行）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Bonn-Rhein-Sieg University of Applied Sciences（波恩-莱茵-锡格应用技术大学）
Recent advances in Explainable AI (XAI) increased the demand for deployment of safe and interpretable AI models in various industry sectors. Despite the latest success of deep neural networks in a variety of domains, understanding the decision-making process of such complex models still remains a challenging task for domain experts. Especially in the financial domain, merely pointing to an anomaly composed of often hundreds of mixed type columns, has limited value for experts. Hence, in this paper, we propose a framework for explaining anomalies using denoising autoencoders designed for mixed …
Explainable Earth Surface Forecasting under Extreme Events
Image Processing Laboratory (IPL), Universitat de València（瓦伦西亚大学图像处理实验室）；Remote Sensing Centre for Earth System Research (RSC4Earth), Leipzig University（莱比锡大学地球系统研究遥感中心）；Institute for Earth System Science and Remote Sensing, Leipzig University（莱比锡大学地球系统科学与遥感研究所）；Image and Signal Processing Group, Leipzig University（莱比锡大学图像与信号处理组）；Helmholtz Centre for Environmental Research – UFZ（亥姆霍兹环境研究中心）
Cybersecurity is a domain where there is constant change in patterns of attack, and we need ways to make our Cybersecurity systems more adaptive to handle new attacks and categorize for appropriate action. We present a novel approach to handle the alarm flooding problem faced by Cybersecurity systems like security information and event management (SIEM) and intrusion detection (IDS). We apply a zero-shot learning method to machine learning (ML) by leveraging explanations for predictions of anomalies generated by a ML model. This approach has huge potential to auto detect alarm labels generated…
Detecting Anomalies in Blockchain Transactions using Machine Learning Classifiers and Explainability Analysis
Premier University（总理大学）；United International University（联合国际大学）；Cyber Security Cooperative Research Centre（网络安全合作研究中心）；Edith Cowan University（埃迪斯科文大学）
Batch Distillation Data for Developing Machine Learning Anomaly Detection Methods
Laboratory of Engineering Thermodynamics, RPTU Kaiserslautern（工程热力学实验室，RPTU凯泽斯劳滕）；Technical University of Munich, Campus Straubing for Biotechnology and Sustainability, Laboratory for Chemical Process Engineering（慕尼黑工业大学，施特劳宾生物技术与可持续发展校区，化学过程工程实验室）；Department of Computer Science, RPTU Kaiserslautern（计算机科学系，RPTU凯泽斯劳滕）
Oak Ridge National Laboratory（橡树岭国家实验室）；North Carolina State University（北卡罗来纳州立大学）
This work presents a novel modeling and analysis framework for graph sequences which addresses the challenge of detecting and contextualizing anomalies in labelled, streaming graph data. We introduce a generalization of the BTER model of Seshadhri et al. by adding flexibility to community structure, and use this model to perform multi-scale graph anomaly detection. Specifically, probability models describing coarse subgraphs are built by aggregating probabilities at finer levels, and these closely related hierarchical models simultaneously detect deviations from expectation. This technique pro…
KU Leuven（鲁汶大学）；Applied Statistics GmbH（应用统计有限公司）；University of Antwerp（安特卫普大学）
A major challenge when trying to detect fraud is that the fraudulent activities form a minority class which make up a very small proportion of the data set. In most data sets, fraud occurs in typically less than 0.5% of the cases. Detecting fraud in such a highly imbalanced data set typically leads to predictions that favor the majority group, causing fraud to remain undetected. We discuss some popular oversampling techniques that solve the problem of imbalanced data by creating synthetic samples that mimic the minority class. A frequent problem when analyzing real data is the presence of anom…
