事后解释方法已成为在高风险应用中理解黑盒分类器的关键工具。然而，高性能分类器通常具有高度非线性，并可能在决策边界附近表现出复杂行为，从而导致局部解释脆弱或具有误导性。因此，迫切需要量化此类解释方法的不确定性，以便理解解释何时可信。在本工作中，我们提出了 Gaussian Process Explanation UnCertainty（GPEC）框架，该框架生成统一的不确定性估计，将决策边界感知不确定性与解释函数逼近不确定性相结合。我们引入了一种新颖的基于测地线的核，用于捕获目标黑盒决策边界的复杂性。我们从理论上证明，所提出的核相似性会随决策边界复杂性的增加而增加。所提出的框架具有高度灵活性；它可与任何黑盒分类器和特征归因方法一起使用。在多个表格和图像数据集上的实证结果表明，与现有方法相比，GPEC 不确定性估计改善了对解释的理解。
英文摘要：
Post-hoc explanation methods have become a critical tool for understanding black-box classifiers in high-stakes applications. However, high-performing classifiers are often highly nonlinear and can exhibit complex behavior around the decision boundary, leading to brittle or misleading local explanations. Therefore there is an impending need to quantify the uncertainty of such explanation methods in order to understand when explanations are trustworthy. In this work we propose the Gaussian Process Explanation UnCertainty (GPEC) framework, which generates a unified uncertainty estimate combining decision boundary-aware uncertainty with explanation function approximation uncertainty. We introduce a novel geodesic-based kernel, which captures the complexity of the target black-box decision boundary. We show theoretically that the proposed kernel similarity increases with decision boundary complexity. The proposed framework is highly flexible; it can be used with any black-box classifier and feature attribution method. Empirical results on multiple tabular and image datasets show that the GPEC uncertainty estimate improves understanding of explanations as compared to existing methods.
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technische Universität Berlin (TU Berlin)（柏林工业大学）
Amid a discussion about Green AI in which we see explainability neglected, we explore the possibility to efficiently approximate computationally expensive explainers. To this end, we propose feature attribution modelling with Empirical Explainers. Empirical Explainers learn from data to predict the attribution maps of expensive explainers. We train and test Empirical Explainers in the language domain and find that they model their expensive counterparts surprisingly well, at a fraction of the cost. They could thus mitigate the computational burden of neural explanations significantly, in appli…
TVE: Learning Meta-attribution for Transferable Vision Explainer
Rice University（莱斯大学）；Wake Forest University（维克森林大学）；New Jersey Institute of Technology（新泽西理工学院）；Texas A&M University（德克萨斯农工大学）；Stevens Institute of Technology（斯蒂文斯理工学院）；North Carolina State University（北卡罗来纳州立大学）；Meta Platforms, Inc.（元平台公司）
University of Science and Technology of China（中国科学技术大学）；CCCD Key Lab of Ministry of Culture and Tourism（文化和旅游部CCCD重点实验室）；National University of Singapore（新加坡国立大学）
Explainability is crucial for probing graph neural networks (GNNs), answering questions like "Why the GNN model makes a certain prediction?". Feature attribution is a prevalent technique of highlighting the explanatory subgraph in the input graph, which plausibly leads the GNN model to make its prediction. Various attribution methods exploit gradient-like or attention scores as the attributions of edges, then select the salient edges with top attribution scores as the explanation. However, most of these works make an untenable assumption - the selected edges are linearly independent - thus lea…
As neural networks increasingly make critical decisions in high-stakes settings, monitoring and explaining their behavior in an understandable and trustworthy manner is a necessity. One commonly used type of explainer is post hoc feature attribution, a family of methods for giving each feature in an input a score corresponding to its influence on a model's output. A major limitation of this family of explainers in practice is that they can disagree on which features are more important than others. Our contribution in this paper is a method of training models with this disagreement problem in m…
Monash University Malaysia（莫纳什大学马来西亚分校）；Monash University（莫纳什大学）
GAN-generated deepfakes as a genre of digital images are gaining ground as both catalysts of artistic expression and malicious forms of deception, therefore demanding systems to enforce and accredit their ethical use. Existing techniques for the source attribution of synthetic images identify subtle intrinsic fingerprints using multiclass classification neural nets limited in functionality and scalability. Hence, we redefine the deepfake detection and source attribution problems as a series of related binary classification tasks. We leverage transfer learning to rapidly adapt forgery detection…
University of Toronto（多伦多大学）；University of Cambridge（剑桥大学）
Recent work has suggested post-hoc explainers might be ineffective for detecting spurious correlations in Deep Neural Networks (DNNs). However, we show there are serious weaknesses with the existing evaluation frameworks for this setting. Previously proposed metrics are extremely difficult to interpret and are not directly comparable between explainer methods. To alleviate these constraints, we propose a new evaluation methodology, Explainer Divergence Scores (EDS), grounded in an information theory approach to evaluate explainers. EDS is easy to interpret and naturally comparable across expla…
Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes
Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands（荷兰阿姆斯特丹自由大学计算机科学系）；Rational Intelligence Lab, CISPA Helmholtz Center for Information Security, Germany（德国信息安全部CISPA理性智能实验室）；College of Computing & Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）
Towards Desiderata-Driven Design of Visual Counterfactual Explainers
Berlin Institute for the Foundations of Learning（柏林学习与数据基础研究所）；Department of Artificial Intelligence, Korea University, Seoul, Korea（人工智能系，韩国大学，首尔，韩国）；Max-Planck Institute for Informatics, Saarbruecken, Germany（马克斯·普朗克信息研究所，萨尔布吕肯，德国）
Feature attribution methods are popular for explaining neural network predictions, and they are often evaluated on metrics such as comprehensiveness and sufficiency. In this paper, we highlight an intriguing property of these metrics: their solvability. Concretely, we can define the problem of optimizing an explanation for a metric, which can be solved by beam search. This observation leads to the obvious yet unaddressed question: why do we use explainers (e.g., LIME) not based on solving the target metric, if the metric value represents explanation quality? We present a series of investigatio…
TU Dresden（德累斯顿工业大学）；University of Bristol（布里斯托大学）
Explainability of black-box machine learning models is crucial, in particular when deployed in critical applications such as medicine or autonomous cars. Existing approaches produce explanations for the predictions of models, however, how to assess the quality and reliability of such explanations remains an open question. In this paper we take a step further in order to provide the practitioner with tools to judge the trustworthiness of an explanation. To this end, we produce estimates of the uncertainty of a given explanation by measuring the ordinal consensus amongst a set of diverse bootstr…
A Unified Framework for Uncertainty-Aware Explainable Artificial Intelligence: A Case Study in Power Quality Disturbance Classification
School of Engineering, Deakin University（德肯大学工程学院）；Faculty of Mathematics and Computer Science, FernUniversität in Hagen（哈根应用科学大学数学与计算机科学学院）；Department of Computing Technologies, Swinburne University of Technology（斯威本科技大学计算技术学院）
The Hong Kong Polytechnic University（香港理工大学）；University of Toronto（多伦多大学）
This paper presents Gem, a model-agnostic approach for providing interpretable explanations for any GNNs on various graph learning tasks. Specifically, we formulate the problem of providing explanations for the decisions of GNNs as a causal learning task. Then we train a causal explanation model equipped with a loss function based on Granger causality. Different from existing explainers for GNNs, Gem explains GNNs on graph-structured data from a causal perspective. It has better generalization ability as it has no requirements on the internal structure of the GNNs or prior knowledge on the gra…
No Single Metric Tells the Whole Story: A Multi-Dimensional Evaluation Framework for Uncertainty Attributions
XITASO GmbH IT & Software Solutions（XITASO GmbH信息技术与软件解决方案）；The Artificial Intelligence and Cognitive Load Research Lab（人工智能与认知负荷研究实验室）；University College Cork（科尔切斯特大学）；Berliner Hochschule für Technik Berlin（柏林技术高等学院）；Faculty of Technology, Policy and Management（技术、政策与管理学院）；Delft University of Technology（代尔夫特理工大学）；School of Computer Science and Information Technology（计算机科学与信息科技学院）
Feature attribution is widely used in interpretable machine learning to explain how influential each measured input feature value is for an output inference. However, measurements can be uncertain, and it is unclear how the awareness of input uncertainty can affect the trust in explanations. We propose and study two approaches to help users to manage their perception of uncertainty in a model explanation: 1) transparently show uncertainty in feature attributions to allow users to reflect on, and 2) suppress attribution to features with uncertain measurements and shift attribution to other feat…
Feedzai（Feedzai公司）；Instituto Superior Técnico, ULisboa（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究所）
Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predicti…
Propagation and Attribution of Uncertainty in Medical Imaging Pipelines
Technical University of Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）；German Cancer Consortium DKTK（德国癌症研究联盟（DKTK））；Helmholtz-Zentrum Munich（慕尼黑亥姆霍兹中心）
Explaining Time Series Classifiers with PHAR: Rule Extraction and Fusion from Post-hoc Attributions
Jagiellonian Human-Centered AI Lab, Mark Kac Center for Complex Systems Research（杰克利昂人类中心人工智能实验室、马克·卡复杂系统研究中心）；Institute of Applied Computer Science, Jagiellonian University（应用计算机科学研究所、杰克利昂大学）
Towards A Unified Information Bottleneck Framework for Time Series Explanations
Florida International University（佛罗里达国际大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Houston（休斯顿大学）；NEC Labs America（美国 NEC 实验室）；Singapore Management University（新加坡管理大学）
Decoding the Past: An Uncertainty-Aware Deep Learning Framework for Sex Attribution in Prehistoric Hand Stencils
Data Science and AI Division, Azyri（Azyri公司数据科学与人工智能部门）；Universidad Autónoma de Ciudad Juárez（华雷斯自治大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Universitat Jaume I（豪梅一世大学）
A Hierarchy of Entropy-Shapley Games for Multivariate Predictive Uncertainty
Leibniz Institute for Prevention Research and Epidemiology – BIPS（莱布尼茨预防研究与流行病学研究所——BIPS）；University of Bremen（不来梅大学）；Simula Research Laboratory（西穆拉研究实验室）；Telenor Research & Innovation（挪威电信研究与创新部）；Norwegian Computing Center（挪威计算中心）
