Game-theoretic formulations of feature importance have become popular as a way to "explain" machine learning models. These methods define a cooperative game between the features of a model and distribute influence among these input elements using some form of the game's unique Shapley values. Justification for these methods rests on two pillars: their desirable mathematical properties, and their applicability to specific motivations for explanations. We show that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity, such as the need for causal reasoning. We also draw on additional literature to argue that Shapley values do not provide explanations which suit human-centric goals of explainability.
A number of techniques have been proposed to explain a machine learning model's prediction by attributing it to the corresponding input features. Popular among these are techniques that apply the Shapley value method from cooperative game theory. While existing papers focus on the axiomatic motivation of Shapley values, and efficient techniques for computing them, they offer little justification for the game formulations used, and do not address the uncertainty implicit in their methods' outputs. For instance, the popular SHAP algorithm's formulation may give substantial attributions to featur…
We introduce a variable importance measure to quantify the impact of individual input variables to a black box function. Our measure is based on the Shapley value from cooperative game theory. Many measures of variable importance operate by changing some predictor values with others held fixed, potentially creating unlikely or even logically impossible combinations. Our cohort Shapley measure uses only observed data points. Instead of changing the value of a predictor we include or exclude subjects similar to the target subject on that predictor to form a similarity cohort. Then we apply Shapl…
Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes
Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands（荷兰阿姆斯特丹自由大学计算机科学系）；Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Germany（德国信息安全部CISPA理性智能实验室）；College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；Google Inc.（谷歌公司）
A popular explainable AI (XAI) approach to quantify feature importance of a given model is via Shapley values. These Shapley values arose in cooperative games, and hence a critical ingredient to compute these in an XAI context is a so-called value function, that computes the "value" of a subset of features, and which connects machine learning models to cooperative games. There are many possible choices for such value functions, which broadly fall into two categories: on-manifold and off-manifold value functions, which take an observational and an interventional viewpoint respectively. Both the…
Shapley values underlie one of the most popular model-agnostic methods within explainable artificial intelligence. These values are designed to attribute the difference between a model's prediction and an average baseline to the different features used as input to the model. Being based on solid game-theoretic principles, Shapley values uniquely satisfy several desirable properties, which is why they are increasingly used to explain the predictions of possibly complex and highly non-linear machine learning models. Shapley values are well calibrated to a user's intuition when features are indep…
Shapley values have become one of the most popular feature attribution explanation methods. However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training. Thus, we propose to incorporate Shapley values themselves as latent representations in deep models thereby making Shapley explanations first-class citizens in the modeling paradigm. This intrinsic explanation approach enables layer-wise explanations, explanation regularization …
What is the value of an individual model in an ensemble of binary classifiers? We answer this question by introducing a class of transferable utility cooperative games called \textit{ensemble games}. In machine learning ensembles, pre-trained models cooperate to make classification decisions. To quantify the importance of models in these ensemble games, we define \textit{Troupe} -- an efficient algorithm which allocates payoffs based on approximate Shapley values of the classifiers. We argue that the Shapley value of models in these games is an effective decision metric for choosing a high per…
Raptor Financial Technologies（Raptor金融技术公司）；Birkbeck College University of London（伦敦大学伯克贝克学院）；University of Birmingham（伯明翰大学）；Tokyo, Japan（东京，日本）
The Shapley value is one of the most widely used measures of feature importance partly as it measures a feature's average effect on a model's prediction. We introduce joint Shapley values, which directly extend Shapley's axioms and intuitions: joint Shapley values measure a set of features' average contribution to a model's prediction. We prove the uniqueness of joint Shapley values, for any order of explanation. Results for games show that joint Shapley values present different insights from existing interaction indices, which assess the effect of a feature within a set of features. The joint…
Even though Shapley value provides an effective explanation for a DNN model prediction, the computation relies on the enumeration of all possible input feature coalitions, which leads to the exponentially growing complexity. To address this problem, we propose a novel method SHEAR to significantly accelerate the Shapley explanation for DNN models, where only a few coalitions of input features are involved in the computation. The selection of the feature coalitions follows our proposed Shapley chain rule to minimize the absolute error from the ground-truth Shapley values, such that the computat…
Explainability in AI is crucial for model development, compliance with regulation, and providing operational nuance to predictions. The Shapley framework for explainability attributes a model's predictions to its input features in a mathematically principled and model-agnostic way. However, general implementations of Shapley explainability make an untenable assumption: that the model's features are uncorrelated. In this work, we demonstrate unambiguous drawbacks of this assumption and develop two solutions to Shapley explainability that respect the data manifold. One solution, based on generat…
The Shapley value has become a popular method to attribute the prediction of a machine-learning model on an input to its base features. The use of the Shapley value is justified by citing [16] showing that it is the \emph{unique} method that satisfies certain good properties (\emph{axioms}). There are, however, a multiplicity of ways in which the Shapley value is operationalized in the attribution problem. These differ in how they reference the model, the training data, and the explanation context. These give very different results, rendering the uniqueness result meaningless. Furthermore, we …
Feature attributions are a common paradigm for model explanations due to their simplicity in assigning a single numeric score for each input feature to a model. In the actionable recourse setting, wherein the goal of the explanations is to improve outcomes for model consumers, it is often unclear how feature attributions should be correctly used. With this work, we aim to strengthen and clarify the link between actionable recourse and feature attributions. Concretely, we propose a variant of SHAP, Counterfactual SHAP (CF-SHAP), that incorporates counterfactual information to produce a backgrou…
Explaining AI systems is fundamental both to the development of high performing models and to the trust placed in them by their users. The Shapley framework for explainability has strength in its general applicability combined with its precise, rigorous foundation: it provides a common, model-agnostic language for AI explainability and uniquely satisfies a set of intuitive mathematical axioms. However, Shapley values are too restrictive in one significant regard: they ignore all causal structure in the data. We introduce a less restrictive framework, Asymmetric Shapley values (ASVs), which are…
DistShap: Scalable GNN Explanations with Distributed Shapley Values
Indiana University Bloomington Indiana, USA；Wake Forest University North Carolina, USA；Texas A\&M University Texas, USA；Indiana University Bloomington；Wake Forest University；Texas A\&M University
University of Toulouse（图卢兹大学）；IRIT（图卢兹计算机科学研究所）；CNRS（法国国家科学研究中心）
This paper develops a rigorous argument for why the use of Shapley values in explainable AI (XAI) will necessarily yield provably misleading information about the relative importance of features for predictions. Concretely, this paper demonstrates that there exist classifiers, and associated predictions, for which the relative importance of features determined by the Shapley values will incorrectly assign more importance to features that are provably irrelevant for the prediction, and less importance to features that are provably relevant for the prediction. The paper also argues that, given r…
School of Mathematics and Physics, The University of Queensland（昆士兰大学数学与物理学院）；SimulaMet, Simula Research Laboratory（SimulaMet、Simula研究院）；Department of Mathematics and Statistics, La Trobe University（拉筹伯大学数学与统计学院）
The Shapley value has become popular in the Explainable AI (XAI) literature, thanks, to a large extent, to a solid theoretical foundation, including four "favourable and fair" axioms for attribution in transferable utility games. The Shapley value is provably the only solution concept satisfying these axioms. In this paper, we introduce the Shapley value and draw attention to its recent uses as a feature selection tool. We call into question this use of the Shapley value, using simple, abstract "toy" counterexamples to illustrate that the axioms may work against the goals of feature selection.…
ShaTS: A Shapley-based Explainability Method for Time Series Artificial Intelligence Models applied to Anomaly Detection in Industrial Internet of Things
Departamento de Ingeniería y Tecnología de Computadores, University of Murcia（计算机工程与技术部门，穆尔西亚大学）
Cohort Shapley value is a model-free method of variable importance grounded in game theory that does not use any unobserved and potentially impossible feature combinations. We use it to evaluate algorithmic fairness, using the well known COMPAS recidivism data as our example. This approach allows one to identify for each individual in a data set the extent to which they were adversely or beneficially affected by their value of a protected attribute such as their race. The method can do this even if race was not one of the original predictors and even if it does not have access to a proprietary…
Here Comes the Explanation: A Shapley Perspective on Multi-contrast Medical Image Segmentation
University of Washington（华盛顿大学）；Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）；Department of Neurological Surgery, University of Washington（华盛顿大学神经外科（注：此处按要求提取到学院/大学级别，实际原文该机构属于华盛顿大学，故提取为华盛顿大学））
Machine learning (ML) and Deep Learning (DL) methods are being adopted rapidly, especially in computer network security, such as fraud detection, network anomaly detection, intrusion detection, and much more. However, the lack of transparency of ML and DL based models is a major obstacle to their implementation and criticized due to its black-box nature, even with such tremendous results. Explainable Artificial Intelligence (XAI) is a promising area that can improve the trustworthiness of these models by giving explanations and interpreting its output. If the internal working of the ML and DL …
Shapley-Value-Based Graph Sparsification for GNN Inference
Department of Intelligent Systems Engineering Indiana University Bloomington Bloomington Indiana USA；Department of Computer Science \& Engineering Texas A\&M University College Station Texas USA；Indiana University Bloomington；Texas A\&M University
ETH Zurich（苏黎世联邦理工学院）；Microsoft（微软）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Amsterdam（阿姆斯特丹大学）
Developing modern machine learning (ML) applications is data-centric, of which one fundamental challenge is to understand the influence of data quality to ML training -- "Which training examples are 'guilty' in making the trained ML model predictions inaccurate or unfair?" Modeling data influence for ML training has attracted intensive interest over the last decade, and one popular framework is to compute the Shapley value of each training example with respect to utilities such as validation accuracy and fairness of the trained ML model. Unfortunately, despite recent intensive interest and res…
Swansea University（斯旺西大学）；University of Plymouth（普利茅斯大学）；Nanyang Technological University（南洋理工大学）
eXplainable Artificial Intelligence (XAI) is a sub-field of Artificial Intelligence (AI) that is at the forefront of AI research. In XAI, feature attribution methods produce explanations in the form of feature importance. People often use feature importance as guidance for intervention. However, a limitation of existing feature attribution methods is that there is a lack of explanation towards the consequence of intervention. In other words, although contribution towards a certain prediction is highlighted by feature attribution methods, the relation between features and the consequence of int…
QGShap: Quantum Acceleration for Faithful GNN Explanations
School of Computer Sciences（计算机科学学院）；National Institute of Science Education and Research（国家科学教育与研究 institute）；An OCC of Homi Bhabha National Institute, India（印度霍米·巴哈瓦国家研究所的分支）
Beyond Shapley: Efficient Computation of Asymmetric Shapley Values
Departamento de Computación, Facultad de Ciencias Exactas y Naturales, UBA（布宜诺斯艾利斯大学自然科学与精确科学学院计算机科学系）；Instituto de Ciencias de la Computación (ICC), CONICET（国家科学与技术研究理事会计算机科学研究所）
IMFD Chile（智利基础数据研究所）；Inria Lille – Nord Europe（法国国家信息与自动化研究所里尔-北欧分部）；UMR 9189 CRIStAL（CRIStAL联合研究单元）
Scores based on Shapley values are widely used for providing explanations to classification results over machine learning models. A prime example of this is the influential SHAP-score, a version of the Shapley value that can help explain the result of a learned model on a specific entity by assigning a score to every feature. While in general computing Shapley values is a computationally intractable problem, it has recently been claimed that the SHAP-score can be computed in polynomial time over the class of decision trees. In this paper, we provide a proof of a stronger result over Boolean mo…
