As machine learning and algorithmic decision making systems are increasingly being leveraged in high-stakes human-in-the-loop settings, there is a pressing need to understand the rationale of their predictions. Researchers have responded to this need with explainable AI (XAI), but often proclaim interpretability axiomatically without evaluation. When these systems are evaluated, they are often tested through offline simulations with proxy metrics of interpretability (such as model complexity). We empirically evaluate the veracity of three common interpretability assumptions through a large scale human-subjects experiment with a simple "placebo explanation" control. We find that feature attribution explanations provide marginal utility in our task for a human decision maker and in certain cases result in worse decisions due to cognitive and contextual confounders. This result challenges the assumed universal benefit of applying these methods and we hope this work will underscore the importance of human evaluation in XAI research. Supplemental materials -- including anonymized data from the experiment, code to replicate the study, an interactive demo of the experiment, and the models used in the analysis -- can be found at: https://doi.pizza/challenging-xai.
Attributions are a common local explanation technique for deep learning models on single samples as they are easily extractable and demonstrate the relevance of input values. In many cases, heatmaps visualize such attributions for samples, for instance, on images. However, heatmaps are not always the ideal visualization to explain certain model decisions for other data types. In this review, we focus on attribution visualizations for time series. We collect attribution heatmap visualizations and some alternatives, discuss the advantages as well as disadvantages and give a short position toward…
Microsoft Research India（微软印度研究院）；University of Chicago（芝加哥大学）
Feature attributions and counterfactual explanations are popular approaches to explain a ML model. The former assigns an importance score to each input feature, while the latter provides input examples with minimal changes to alter the model's predictions. To unify these approaches, we provide an interpretation based on the actual causality framework and present two key results in terms of their use. First, we present a method to generate feature attribution explanations from a set of counterfactual examples. These feature attributions convey how important a feature is to changing the classifi…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；TU Kaiserslautern（凯泽斯劳滕工业大学）
With the rise in the employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, time-series data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in a novel way by proposing TSInsight where we attach an auto-encoder to the classifier with a sparsity-inducing norm on its output and fine-tune it based on the gradients from the classifier and a recon…
The Metagame of Interpretability and Meta-Attributions
University of Warsaw（华沙大学）；Centre for Credible AI, Warsaw University of Technology（可信AI中心，华沙技术大学）；Bielefeld University（比勒菲尔德大学）；LMU Munich, MCML（慕尼黑大学LMU，MCML）
Adobe（奥多比公司）；Carnegie Mellon University（卡内基梅隆大学）；University of Tübingen（蒂宾根大学）；Harvard University（哈佛大学）
As attribution-based explanation methods are increasingly used to establish model trustworthiness in high-stakes situations, it is critical to ensure that these explanations are stable, e.g., robust to infinitesimal perturbations to an input. However, previous works have shown that state-of-the-art explanation methods generate unstable explanations. Here, we introduce metrics to quantify the stability of an explanation and show that several popular explanation methods are unstable. In particular, we propose new Relative Stability metrics that measure the change in output explanation with respe…
Feature attribution is widely used in interpretable machine learning to explain how influential each measured input feature value is for an output inference. However, measurements can be uncertain, and it is unclear how the awareness of input uncertainty can affect the trust in explanations. We propose and study two approaches to help users to manage their perception of uncertainty in a model explanation: 1) transparently show uncertainty in feature attributions to allow users to reflect on, and 2) suppress attribution to features with uncertain measurements and shift attribution to other feat…
Advancing climate model interpretability: Feature attribution for Arctic melt anomalies
Institute for Harnessing Data and Model Revolution in the Polar Regions（极地地区数据与模型利用研究所）；University of Maryland, Baltimore County（马里兰大学巴尔的摩分校）；National Oceanic and Atmospheric Administration（美国国家海洋和大气管理局）
Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（香港城市大学计算机科学系）；Harbin Institute of Technology, Harbin, China（哈尔滨工业大学）
School of Computer Science, Peking University, Beijing 100871, China（北京大学计算机科学学院）；Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术重点实验室（北京大学））
Pittsburgh, PA 15213, USA（美国宾夕法尼亚州匹兹堡（注：该地址不属于机构，可能是原数据错误，按要求应忽略，此处仅为示例说明））
Recent work has found that adversarially-robust deep networks used for image classification are more interpretable: their feature attributions tend to be sharper, and are more concentrated on the objects associated with the image's ground-truth class. We show that smooth decision boundaries play an important role in this enhanced interpretability, as the model's input gradients around data points will more closely align with boundaries' normal vectors when they are smooth. Thus, because robust models have smoother boundaries, the results of gradient-based attribution methods, like Integrated G…
Attribution methods have been shown as promising approaches for identifying key features that led to learned model predictions. While most existing attribution methods rely on a baseline input for performing feature perturbations, limited research has been conducted to address the baseline selection issues. Poor choices of baselines limit the ability of one-vs-one (1-vs-1) explanations for multi-class classifiers, which means the attribution methods were not able to explain why an input belongs to its original class but not the other specified target class. 1-vs-1 explanation is crucial when c…
SKEMA Business School（SKEMA商学院）；Millennium Institute for Foundations of Data Research（千禧数据基础研究所）
In this expository article we highlight the relevance of explanations for artificial intelligence, in general, and for the newer developments in {\em explainable AI}, referring to origins and connections of and among different approaches. We describe in simple terms, explanations in data management and machine learning that are based on attribution-scores, and counterfactuals as found in the area of causality. We elaborate on the importance of logical reasoning when dealing with counterfactuals, and their use for score computation.
IBM Research（IBM研究院）；Rensselaer Polytechnic Institute（伦斯勒理工学院）
Post-hoc explanations for black box models have been studied extensively in classification and regression settings. However, explanations for models that output similarity between two inputs have received comparatively lesser attention. In this paper, we provide model agnostic local explanations for similarity learners applicable to tabular and text data. We first propose a method that provides feature attributions to explain the similarity between a pair of inputs as determined by a black box similarity learner. We then propose analogies as a new form of explanation in machine learning. Here …
Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only data
Oslo University Hospital（奥斯陆大学医院）；University of Oslo（奥斯陆大学）；Johannes Kepler University Linz（约翰·开普勒林茨大学）；Novo Nordisk A/S（诺和诺德公司）；University Hospital Basel（巴塞尔大学医院）；Imprint Labs, LLC.（Imprint Labs有限责任公司）
Provably Better Explanations with Optimized Aggregation of Feature Attributions
LMU Munich（慕尼黑大学）；Siemens AG（西门子公司）；Technical University of Munich（慕尼黑工业大学）；University of Oxford（牛津大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Goethe University Frankfurt（法兰克福大学）
Local explanation methods, also known as attribution methods, attribute a deep network's prediction to its input (cf. Baehrens et al. (2010)). We respond to the claim from Adebayo et al. (2018) that local explanation methods lack sensitivity, i.e., DNNs with randomly-initialized weights produce explanations that are both visually and quantitatively similar to those produced by DNNs with learned weights. Further investigation reveals that their findings are due to two choices in their analysis: (a) ignoring the signs of the attributions; and (b) for integrated gradients (IG), including pixels i…
Interpretability is an important area of research for safe deployment of machine learning systems. One particular type of interpretability method attributes model decisions to input features. Despite active development, quantitative evaluation of feature attribution methods remains difficult due to the lack of ground truth: we do not know which input features are in fact important to a model. In this work, we propose a framework for Benchmarking Attribution Methods (BAM) with a priori knowledge of relative feature importance. BAM includes 1) a carefully crafted dataset and models trained with …
Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them. This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori. We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages. Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution …
University of California, Irvine（加州大学欧文分校）；Harvard University（哈佛大学）
As machine learning models are increasingly used in critical decision-making settings (e.g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions. Such \textit{explanations} are used to understand and establish trust in models and are vital components in machine learning pipelines. Though explanations are a critical piece in these systems, there is little understanding about how they are vulnerable to manipulation by adversaries. In this paper, we discuss how two broad classes of explanations are vulnerable to manipulation. We demonstrate h…
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present Summit, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has lear…
Comprehensive and Reliable Feature Attribution for Diverse Modalities and Models via Frequency-Domain Insights
School of Computational Science, Wuhan University（武汉大学计算科学学院）；Brain Research Center, Wuhan University（武汉大学脑科学研究中心）；College of Information Science and Technology (School of Cyber Science and Technology), Shihezi University（石河子大学信息科学学院（网络安全科学与技术学院））；Xinjiang Production and Construction Corps Key Laboratory of Computing Intelligence and Network Information Security Open Fund（新疆生产建设兵团计算智能与网络信息安全重点实验室开放基金）
The Pingry School（平格里学校）；Chinle Comprehensive Health Care Facility（钦利综合医疗中心）；University of Maryland, College Park（马里兰大学帕克分校）；NVIDIA（英伟达）
30 million Optical Coherence Tomography (OCT) imaging tests are issued annually to diagnose various retinal diseases, but accurate diagnosis of OCT scans requires trained eye care professionals who are still prone to making errors. With better systems for diagnosis, many cases of vision loss caused by retinal disease could be entirely avoided. In this work, we present ReLaX, a novel deep learning framework for explainable, accurate classification of retinal pathologies which achieves state-of-the-art accuracy. Furthermore, we emphasize producing both qualitative and quantitative explanations o…
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）
Recent research has demonstrated that feature attribution methods for deep networks can themselves be incorporated into training; these attribution priors optimize for a model whose attributions have certain desirable properties -- most frequently, that particular features are important or unimportant. These attribution priors are often based on attribution methods that are not guaranteed to satisfy desirable interpretability axioms, such as completeness and implementation invariance. Here, we introduce attribution priors to optimize for higher-level properties of explanations, such as smoothn…
CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Jinesis AI Lab（Jinesis人工智能实验室）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；CMU（卡内基梅隆大学）；EuroSafeAI；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Minimizing False-Positive Attributions in Explanations of Non-Linear Models
Technical University of Denmark（丹麦技术大学）；Technische Universität Berlin（柏林技术大学）；Physikalisch-Technische Bundesanstalt, Berlin（柏林物理技术部）；Charité – Universitätsmedizin Berlin（柏林查理医院大学医学中心）
Auburn University（奥本大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Attribution methods can provide powerful insights into the reasons for a classifier's decision. We argue that a key desideratum of an explanation method is its robustness to input hyperparameters which are often randomly set or empirically tuned. High sensitivity to arbitrary hyperparameter choices does not only impede reproducibility but also questions the correctness of an explanation and impairs the trust of end-users. In this paper, we provide a thorough empirical study on the sensitivity of existing attribution methods. We found an alarming trend that many methods are highly sensitive to …
XaiPient；University of Wisconsin (Madison)（威斯康星大学麦迪逊分校）；Google（谷歌）
We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions. Two desirable characteristics of an attribution-based explanation are: (1) $\textit{sparseness}$: the attributions of irrelevant or weakly relevant features should be negligible, thus resulting in $\textit{concise}$ explanations in terms of the significant features, and (2) $\textit{stability}$: it should not vary significantly within a small local neighborhoo…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）
While rule-based attribution methods have proven useful for providing local explanations for Deep Neural Networks, explaining modern and more varied network architectures yields new challenges in generating trustworthy explanations, since the established rule sets might not be sufficient or applicable to novel network structures. As an elegant solution to the above issue, network canonization has recently been introduced. This procedure leverages the implementation-dependency of rule-based attributions and restructures a model into a functionally identical equivalent of alternative design to w…
Ghent University（根特大学）；University of Edinburgh（爱丁堡大学）
Even though deep neural networks (DNNs) achieve state-of-the-art results for a number of problems involving genomic data, getting DNNs to explain their decision-making process has been a major challenge due to their black-box nature. One way to get DNNs to explain their reasoning for prediction is via attribution methods which are assumed to highlight the parts of the input that contribute to the prediction the most. Given the existence of numerous attribution methods and a lack of quantitative results on the fidelity of those methods, selection of an attribution method for sequence-based task…
Northwestern University（西北大学）；Ege University（爱琴海大学）；Tokat State Hospital（托卡特国立医院）；NVIDIA（英伟达）；University of Catania（卡塔尼亚大学）；Columbia University Medical Center（哥伦比亚大学医学中心）；National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所）
Visual explanation methods have an important role in the prognosis of the patients where the annotated data is limited or unavailable. There have been several attempts to use gradient-based attribution methods to localize pathology from medical scans without using segmentation labels. This research direction has been impeded by the lack of robustness and reliability. These methods are highly sensitive to the network parameters. In this study, we introduce a robust visual explanation method to address this problem for medical applications. We provide an innovative visual explanation algorithm f…
The majority of existing post-hoc explanation approaches for machine learning models produce independent, per-variable feature attribution scores, ignoring a critical inherent characteristics of homogeneously structured data, such as visual or text data: there exist latent inter-variable relationships between features. In response, we develop a novel model-agnostic and permutation-based feature attribution approach based on the relational analysis between input variables. As a result, we are able to gain a broader insight into the predictions and decisions of machine learning models. Experimen…
IBM Research Zurich（IBM苏黎世研究院）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Explanations are crucial parts of deep neural network (DNN) classifiers. In high stakes applications, faithful and robust explanations are important to understand and gain trust in DNN classifiers. However, recent work has shown that state-of-the-art attribution methods in text classifiers are susceptible to imperceptible adversarial perturbations that alter explanations significantly while maintaining the correct prediction outcome. If undetected, this can critically mislead the users of DNNs. Thus, it is crucial to understand the influence of such adversarial perturbations on the networks' e…
Feature attribution methods, which explain an individual prediction made by a model as a sum of attributions for each input feature, are an essential tool for understanding the behavior of complex deep learning models. However, ensuring that models produce meaningful explanations, rather than ones that rely on noise, is not straightforward. Exacerbating this problem is the fact that attribution methods do not provide insight as to why features are assigned their attribution values, leading to explanations that are difficult to interpret. In real-world problems we often have sets of additional …
