We compare machine learning explainability methods based on the theory of atomic (Shapley, 1953) and infinitesimal (Aumann and Shapley, 1974) games, in a theoretical and experimental investigation into how the model and choice of integration path can influence the resulting feature attributions. To gain insight into differences in attributions resulting from interventional Shapley values (Sundararajan and Najmi, 2019; Janzing et al., 2019; Chen et al., 2019) and Generalized Integrated Gradients (GIG) (Merrill et al., 2019) we note interventional Shapley is equivalent to a multi-path integration along $n!$ paths where $n$ is the number of model input features. Applying Stoke's theorem we show that the path symmetry of these two methods results in the same attributions when the model is composed of a sum of separable functions of individual features and a sum of two-feature products. We then perform a series of experiments with varying degrees of data missingness to demonstrate how interventional Shapley's multi-path approach can yield less consistent attributions than the single straight-line path of Aumann-Shapley. We argue this is because the multiple paths employed by interventional Shapley extend away from the training data manifold and are therefore more likely to pass through regions where the model has little support. In the absence of a more meaningful path choice, we therefore advocate the straight-line path since it will almost always pass closer to the data manifold. Among straight-line path attribution algorithms, GIG is uniquely robust since it will still yield Shapley values for atomic games modeled by decision trees.
The Metagame of Interpretability and Meta-Attributions
University of Warsaw（华沙大学）；Centre for Credible AI, Warsaw University of Technology（可信AI中心，华沙技术大学）；Bielefeld University（比勒菲尔德大学）；LMU Munich, MCML（慕尼黑大学LMU，MCML）
MeClear: Cooperative Game-Theoretic Attribution and Risk-Aware Memory Clearance for Long-Horizon LLM Agents
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机与人工智能学院）；School of Cyberspace Science and Technology, Beijing Institute of Technology（北京理工大学网络空间科学与技术学院）
Who Does What in Deep Learning? Multidimensional Game-Theoretic Attribution of Function of Neural Units
Institute of Computational Neuroscience（计算神经科学研究所）；University Medical Center Eppendorf（埃本多夫大学医学中心）；International Max Planck Research School on Cognitive Neuroimaging（认知神经成像国际马克斯·普朗克研究学校）；MRC Cognition and Brain Sciences Unit（MRC认知与脑科学单位）；Mila - Quebec Artificial Intelligence Institute（魁北克人工智能研究所）；Learning in Machines & Brains（机器与大脑学习）；Departments of Bioengineering and Neuroscience（生物工程与神经科学系）；Department of Health Sciences（健康科学系）
XaiPient；University of Wisconsin (Madison)（威斯康星大学麦迪逊分校）；Google（谷歌）
We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions. Two desirable characteristics of an attribution-based explanation are: (1) $\textit{sparseness}$: the attributions of irrelevant or weakly relevant features should be negligible, thus resulting in $\textit{concise}$ explanations in terms of the significant features, and (2) $\textit{stability}$: it should not vary significantly within a small local neighborhoo…
While $\ell_2$ regularization is widely used in training gradient boosted trees, popular individualized feature attribution methods for trees such as Saabas and TreeSHAP overlook the training procedure. We propose Prediction Decomposition Attribution (PreDecomp), a novel individualized feature attribution for gradient boosted trees when they are trained with $\ell_2$ regularization. Theoretical analysis shows that the inner product between PreDecomp and labels on in-sample data is essentially the total gain of a tree, and that it can faithfully recover additive models in the population case wh…
Attribution methods calculate attributions that visually explain the predictions of deep neural networks (DNNs) by highlighting important parts of the input features. In particular, gradient-based attribution (GBA) methods are widely used because they can be easily implemented through automatic differentiation. In this study, we use the attributions that filter out irrelevant parts of the input features and then verify the effectiveness of this approach by measuring the classification accuracy of a pre-trained DNN. This is achieved by calculating and applying an \textit{attribution mask} to th…
Playing the network backward: A Game Theoretic Attribution Framework
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希-赫兹研究所）；Technische Universität Berlin（柏林技术大学）；Freie Universität Berlin（自由大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；Technological University Dublin（都柏林技术大学）
Feature attribution (FA), or the assignment of class-relevance to different locations in an image, is important for many classification problems but is particularly crucial within the neuroscience domain, where accurate mechanistic models of behaviours, or disease, require knowledge of all features discriminative of a trait. At the same time, predicting class relevance from brain images is challenging as phenotypes are typically heterogeneous, and changes occur against a background of significant natural variation. Here, we present a novel framework for creating class specific FA maps through …
Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI
Tübingen AI Center, University of Tübingen Germany（图宾根人工智能中心、图宾根大学德国）；Fraunhofer Heinrich Hertz Institute Germany（弗劳恩霍夫海因里希·赫兹研究所德国）；Information and Interaction Design, Yonsei University South Korea（信息与交互设计，延世大学韩国）
Additive feature explanations using Shapley values have become popular for providing transparency into the relative importance of each feature to an individual prediction of a machine learning model. While Shapley values provide a unique additive feature attribution in cooperative game theory, the Shapley values that can be generated for even a single machine learning model are far from unique, with theoretical and implementational decisions affecting the resulting attributions. Here, we consider the application of Shapley values for explaining decision tree ensembles and present a novel appro…
Higher-Order Feature Attribution: Bridging Statistics, Explainable AI, and Topological Signal Processing
School of Engineering, The University of Edinburgh（爱丁堡大学工程学院）；Causality in Healthcare AI Hub (CHAI)（医疗AI因果性研究中心）；Department of Electrical and Computer Engineering, Stony Brook University（石溪大学电气与计算机工程系）
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present Summit, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has lear…
Paul G. Allen School of CSE（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
In healthcare, making the best possible predictions with complex models (e.g., neural networks, ensembles/stacks of different models) can impact patient welfare. In order to make these complex models explainable, we present DeepSHAP for mixed model types, a framework for layer wise propagation of Shapley values that builds upon DeepLIFT (an existing approach for explaining neural networks). We show that in addition to being able to explain neural networks, this new framework naturally enables attributions for stacks of mixed models (e.g., neural network feature extractor into a tree model) as …
MFABA: A More Faithful and Accelerated Boundary-based Attribution Method for Deep Neural Networks
The University of Sydney（悉尼大学）；SuZhouYierqi（苏州义尔齐）；University of Malaya（马来亚大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；Wayne State University（韦恩州立大学）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technical University of Berlin（柏林工业大学）；Berlin Institute for the Foundations of Learning and Data (BIFOLD)（柏林学习与数据基础研究所）
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Feature attributions based on the Shapley value are popular for explaining machine learning models; however, their estimation is complex from both a theoretical and computational standpoint. We disentangle this complexity into two factors: (1)~the approach to removing feature information, and (2)~the tractable estimation strategy. These two factors provide a natural lens through which we can better understand and compare 24 distinct algorithms. Based on the various feature removal approaches, we describe the multiple types of Shapley value feature attributions and methods to calculate each one…
