A Characteristic Function for Shapley-Value-Based Attribution of Anomaly Scores
University of Applied Sciences and Arts Western Switzerland（瑞士西部应用科学与艺术大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Osaka University（大阪大学）
In anomaly detection, the degree of irregularity is often summarized as a real-valued anomaly score. We address the problem of attributing such anomaly scores to input features for interpreting the results of anomaly detection. We particularly investigate the use of the Shapley value for attributing anomaly scores of semi-supervised detection methods. We propose a characteristic function specifically designed for attributing anomaly scores. The idea is to approximate the absence of some features by locally minimizing the anomaly score with regard to the to-be-absent features. We examine the applicability of the proposed characteristic function and other general approaches for interpreting anomaly scores on multiple datasets and multiple anomaly detection methods. The results indicate the potential utility of the attribution methods including the proposed one.
Center for Advanced Intelligence Project RIKEN（理化学研究所先进智能项目中心）
We present a method to compute the Shapley values of reconstruction errors of principal component analysis (PCA), which is particularly useful in explaining the results of anomaly detection based on PCA. Because features are usually correlated when PCA-based anomaly detection is applied, care must be taken in computing a value function for the Shapley values. We utilize the probabilistic view of PCA, particularly its conditional distribution, to exactly compute a value function for the Shapely values. We also present numerical examples, which imply that the Shapley values are advantageous for …
ShaTS: A Shapley-based Explainability Method for Time Series Artificial Intelligence Models applied to Anomaly Detection in Industrial Internet of Things
Departamento de Ingeniería y Tecnología de Computadores, University of Murcia（计算机工程与技术部门，穆尔西亚大学）
Machine learning (ML) and Deep Learning (DL) methods are being adopted rapidly, especially in computer network security, such as fraud detection, network anomaly detection, intrusion detection, and much more. However, the lack of transparency of ML and DL based models is a major obstacle to their implementation and criticized due to its black-box nature, even with such tremendous results. Explainable Artificial Intelligence (XAI) is a promising area that can improve the trustworthiness of these models by giving explanations and interpreting its output. If the internal working of the ML and DL …
Conditional Attribution for Root Cause Analysis in Time-Series Anomaly Detection
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Paul Wurth S.A（保罗·沃思公司）；Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau（莱茵河-巴尔夫技术大学凯撒斯劳滕-兰道）
We compare machine learning explainability methods based on the theory of atomic (Shapley, 1953) and infinitesimal (Aumann and Shapley, 1974) games, in a theoretical and experimental investigation into how the model and choice of integration path can influence the resulting feature attributions. To gain insight into differences in attributions resulting from interventional Shapley values (Sundararajan and Najmi, 2019; Janzing et al., 2019; Chen et al., 2019) and Generalized Integrated Gradients (GIG) (Merrill et al., 2019) we note interventional Shapley is equivalent to a multi-path integratio…
Google（谷歌）；The George Washington University（乔治华盛顿大学）
The need for explainable AI (XAI) is well established but relatively little has been published outside of the supervised learning paradigm. This paper focuses on a principled approach to applying explainability and interpretability to the task of unsupervised anomaly detection. We argue that explainability is principally an algorithmic task and interpretability is principally a cognitive task, and draw on insights from the cognitive sciences to propose a general-purpose method for practical diagnosis using explained anomalies. We define Attribution Error, and demonstrate, using real-world labe…
Columbia University（哥伦比亚大学）；Stanford University（斯坦福大学）；Amazon AWS（亚马逊云科技）
Shapley value is a popular approach for measuring the influence of individual features. While Shapley feature attribution is built upon desiderata from game theory, some of its constraints may be less natural in certain machine learning settings, leading to unintuitive model interpretation. In particular, the Shapley value uses the same weight for all marginal contributions -- i.e. it gives the same importance when a large number of other features are given versus when a small number of other features are given. This property can be problematic if larger feature sets are more or less informati…
A number of techniques have been proposed to explain a machine learning model's prediction by attributing it to the corresponding input features. Popular among these are techniques that apply the Shapley value method from cooperative game theory. While existing papers focus on the axiomatic motivation of Shapley values, and efficient techniques for computing them, they offer little justification for the game formulations used, and do not address the uncertainty implicit in their methods' outputs. For instance, the popular SHAP algorithm's formulation may give substantial attributions to featur…
Feature attributions are a common paradigm for model explanations due to their simplicity in assigning a single numeric score for each input feature to a model. In the actionable recourse setting, wherein the goal of the explanations is to improve outcomes for model consumers, it is often unclear how feature attributions should be correctly used. With this work, we aim to strengthen and clarify the link between actionable recourse and feature attributions. Concretely, we propose a variant of SHAP, Counterfactual SHAP (CF-SHAP), that incorporates counterfactual information to produce a backgrou…
The Shapley value has become a popular method to attribute the prediction of a machine-learning model on an input to its base features. The use of the Shapley value is justified by citing [16] showing that it is the \emph{unique} method that satisfies certain good properties (\emph{axioms}). There are, however, a multiplicity of ways in which the Shapley value is operationalized in the attribution problem. These differ in how they reference the model, the training data, and the explanation context. These give very different results, rendering the uniqueness result meaningless. Furthermore, we …
Paul G. Allen School of CSE（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
In healthcare, making the best possible predictions with complex models (e.g., neural networks, ensembles/stacks of different models) can impact patient welfare. In order to make these complex models explainable, we present DeepSHAP for mixed model types, a framework for layer wise propagation of Shapley values that builds upon DeepLIFT (an existing approach for explaining neural networks). We show that in addition to being able to explain neural networks, this new framework naturally enables attributions for stacks of mixed models (e.g., neural network feature extractor into a tree model) as …
Additive feature explanations using Shapley values have become popular for providing transparency into the relative importance of each feature to an individual prediction of a machine learning model. While Shapley values provide a unique additive feature attribution in cooperative game theory, the Shapley values that can be generated for even a single machine learning model are far from unique, with theoretical and implementational decisions affecting the resulting attributions. Here, we consider the application of Shapley values for explaining decision tree ensembles and present a novel appro…
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Feature attributions based on the Shapley value are popular for explaining machine learning models; however, their estimation is complex from both a theoretical and computational standpoint. We disentangle this complexity into two factors: (1)~the approach to removing feature information, and (2)~the tractable estimation strategy. These two factors provide a natural lens through which we can better understand and compare 24 distinct algorithms. Based on the various feature removal approaches, we describe the multiple types of Shapley value feature attributions and methods to calculate each one…
Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes
Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands（荷兰阿姆斯特丹自由大学计算机科学系）；Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Germany（德国信息安全部CISPA理性智能实验室）；College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）
Fair feature attribution for multi-output prediction: a Shapley-based perspective
\, Chair of Computational Mathematics, DeustoTech, University of Deusto, Avenida de las Universidades 24, 48007 Bilbao, Basque Country, Spain \, Chair for dynamics, control, machine learning；numerics (Alexander Von Humboldt-Professorship), Department of Mathematics, Friedrich-Alexander-Universit\"at Erlangen-N\"urnberg, 91058, Erlangen, Germany. \, Universidad Aut\'onoma de Madrid, Departamento de Matem\'aticas, Ciudad Universitaria de Cantoblanco, 28049 Madrid, Spain. \, ATLAS Molecular Pharma, Parque Tecnol\'ogico de Bizkaia, Ed. 800, 48160 Derio, Spain. \, Precision Medicine；Metabolism Laboratory, CIC bioGUNE, Basque Research
A Polynomial Architecture-Attribution Co-Design Framework for Exact Aumann-Shapley Attribution in GNNs
Institute of Artificial Intelligence Innovation and Industry, Fudan University（复旦大学人工智能创新与产业研究院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；School of Information and Communication Engineering, Communication University of China（中国传媒大学信息与通信工程学院）
ETH Zurich（苏黎世联邦理工学院）；Disney Research Zurich（苏黎世迪士尼研究中心）
The problem of explaining the behavior of deep neural networks has recently gained a lot of attention. While several attribution methods have been proposed, most come without strong theoretical foundations, which raises questions about their reliability. On the other hand, the literature on cooperative game theory suggests Shapley values as a unique way of assigning relevance scores such that certain desirable properties are satisfied. Unfortunately, the exact evaluation of Shapley values is prohibitively expensive, exponential in the number of input features. In this work, by leveraging recen…
The Metagame of Interpretability and Meta-Attributions
University of Warsaw（华沙大学）；Centre for Credible AI, Warsaw University of Technology（可信AI中心，华沙技术大学）；Bielefeld University（比勒菲尔德大学）；LMU Munich, MCML（慕尼黑大学LMU，MCML）
Mohamed bin Zayed University of Artificial Intelligence（莫萨德·本·泽德人工智能大学）；University at Buffalo（布法罗大学）；King Abdullah University of Science and Technology（卡布斯国王科学与技术大学）
University of Oxford（牛津大学）；Microsoft Research Cambridge（微软剑桥研究院）
Feature attribution for kernel methods is often heuristic and not individualised for each prediction. To address this, we turn to the concept of Shapley values~(SV), a coalition game theoretical framework that has previously been applied to different machine learning model interpretation tasks, such as linear models, tree ensembles and deep networks. By analysing SVs from a functional perspective, we propose \textsc{RKHS-SHAP}, an attribution method for kernel machines that can efficiently compute both \emph{Interventional} and \emph{Observational Shapley values} using kernel mean embeddings o…
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Local feature attribution methods are increasingly used to explain complex machine learning models. However, current methods are limited because they are extremely expensive to compute or are not capable of explaining a distributed series of models where each model is owned by a separate institution. The latter is particularly important because it often arises in finance where explanations are mandated. Here, we present DeepSHAP, a tractable method to propagate local feature attributions through complex series of models based on a connection to the Shapley value. We evaluate DeepSHAP across bi…
Raptor Financial Technologies（Raptor金融技术公司）；Birkbeck College University of London（伦敦大学伯克贝克学院）；University of Birmingham（伯明翰大学）；Tokyo, Japan（东京，日本）
The Shapley value is one of the most widely used measures of feature importance partly as it measures a feature's average effect on a model's prediction. We introduce joint Shapley values, which directly extend Shapley's axioms and intuitions: joint Shapley values measure a set of features' average contribution to a model's prediction. We prove the uniqueness of joint Shapley values, for any order of explanation. Results for games show that joint Shapley values present different insights from existing interaction indices, which assess the effect of a feature within a set of features. The joint…
School of Mathematics and Physics, The University of Queensland（昆士兰大学数学与物理学院）；SimulaMet, Simula Research Laboratory（SimulaMet、Simula研究院）；Department of Mathematics and Statistics, La Trobe University（拉筹伯大学数学与统计学院）
The Shapley value has become popular in the Explainable AI (XAI) literature, thanks, to a large extent, to a solid theoretical foundation, including four "favourable and fair" axioms for attribution in transferable utility games. The Shapley value is provably the only solution concept satisfying these axioms. In this paper, we introduce the Shapley value and draw attention to its recent uses as a feature selection tool. We call into question this use of the Shapley value, using simple, abstract "toy" counterexamples to illustrate that the axioms may work against the goals of feature selection.…
Energy-Based Model for Accurate Estimation of Shapley Values in Feature Attribution
College of Metrology Measurement and Instrument, China Jiliang University（计量测量与仪器学院，中国浙江理工大学）；School of Mathematics, Hangzhou Normal University（数学学院，杭州师范大学）；School of Statistics, Jiangxi University of Finance and Economics（统计学院，江西财经大学）
A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty
Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱布尼茨预防研究与流行病学研究所——BIPS）；University of Bremen（不来梅大学）；Simula Research Laboratory（西穆拉研究实验室）；Telenor Research & Innovation（挪威电信研究与创新部）；Norwegian Computing Center（挪威计算中心）
Shapley values, which were originally designed to assign attributions to individual players in coalition games, have become a commonly used approach in explainable machine learning to provide attributions to input features for black-box machine learning models. A key attraction of Shapley values is that they uniquely satisfy a very natural set of axiomatic properties. However, extending the Shapley value to assigning attributions to interactions rather than individual players, an interaction index, is non-trivial: as the natural set of axioms for the original Shapley values, extended to the co…
This paper aims to explain deep neural networks (DNNs) from the perspective of multivariate interactions. In this paper, we define and quantify the significance of interactions among multiple input variables of the DNN. Input variables with strong interactions usually form a coalition and reflect prototype features, which are memorized and used by the DNN for inference. We define the significance of interactions based on the Shapley value, which is designed to assign the attribution value of each input variable to the inference. We have conducted experiments with various DNNs. Experimental res…
VARSHAP: Addressing Global Dependency Problems in Explainable AI with Variance-Based Local Feature Attribution
Faculty of Computing and Telecommunications（计算机与电信学院）；Poznan University of Technology（波兹南技术大学）；Faculty of Mathematics, Informatics and Mechanics（数学、信息学与力学学院）；University of Warsaw（华沙大学）；MIM Solutuions Research Institute（MIM解决方案研究所）
