Attribution methods have been shown as promising approaches for identifying key features that led to learned model predictions. While most existing attribution methods rely on a baseline input for performing feature perturbations, limited research has been conducted to address the baseline selection issues. Poor choices of baselines limit the ability of one-vs-one (1-vs-1) explanations for multi-class classifiers, which means the attribution methods were not able to explain why an input belongs to its original class but not the other specified target class. 1-vs-1 explanation is crucial when certain classes are more similar than others, e.g. two bird types among multiple animals, by focusing on key differentiating features rather than shared features across classes. In this paper, we present GAN-based Model EXplainability (GANMEX), a novel approach applying Generative Adversarial Networks (GAN) by incorporating the to-be-explained classifier as part of the adversarial networks. Our approach effectively selects the counterfactual baseline as the closest realistic sample belong to the target class, which allows attribution methods to provide true 1-vs-1 explanations. We showed that GANMEX baselines improved the saliency maps and led to stronger performance on perturbation-based evaluation metrics over the existing baselines. Existing attribution results are known for being insensitive to model randomization, and we demonstrated that GANMEX baselines led to better outcome under the cascading randomization of the model.
On the notion of missingness for path attribution explainability methods in medical settings: Guiding the selection of medically meaningful baselines
Research Group MITI, Technical University of Munich（慕尼黑技术大学MITI研究组）；Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital（慕尼黑技术大学人工智能在医疗健康中的chair及慕尼黑技术大学医院）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）；Department of Surgery, Technical University of Munich（慕尼黑技术大学外科系）
GUDA: Counterfactual Group-wise Training Data Attribution for Diffusion Models via Unlearning
University of Tokyo（东京大学）；Toyota Central Research Laboratory（丰田中央研究所）；University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；National Institute of Advanced Industrial Science and Technology（国家工业科学与技术研究院）
Model explanations based on pure observational data cannot compute the effects of features reliably, due to their inability to estimate how each factor alteration could affect the rest. We argue that explanations should be based on the causal model of the data and the derived intervened causal models, that represent the data distribution subject to interventions. With these models, we can compute counterfactuals, new samples that will inform us how the model reacts to feature changes on our input. We propose a novel explanation methodology based on Causal Counterfactuals and identify the limit…
Microsoft Research（微软研究院）；Microsoft Bing Ads（微软必应广告）
Given an unexpected change in the output metric of a large-scale system, it is important to answer why the change occurred: which inputs caused the change in metric? A key component of such an attribution question is estimating the counterfactual: the (hypothetical) change in the system metric due to a specified change in a single input. However, due to inherent stochasticity and complex interactions between parts of the system, it is difficult to model an output metric directly. We utilize the computational structure of a system to break up the modelling task into sub-parts, such that each su…
There exists an apparent negative correlation between performance and interpretability of deep learning models. In an effort to reduce this negative correlation, we propose a Born Identity Network (BIN), which is a post-hoc approach for producing multi-way counterfactual maps. A counterfactual map transforms an input sample to be conditioned and classified as a target label, which is similar to how humans process knowledge through counterfactual thinking. For example, a counterfactual map can localize hypothetical abnormalities from a normal brain image that may cause it to be diagnosed with a…
Tracing Generated Samples to Training-Data Clusters in Flow-Matching Models
Forschungszentrum Jülich（于利希研究中心）；Technical University Dortmund（多特蒙德工业大学）；Reichman University（赖希曼大学）；Lamarr Institute（拉马尔研究所）；Institute for AI in Medicine, University Hospital Essen（埃森大学医院医学人工智能研究所）；Helmholtz AI（亥姆霍兹人工智能）；University of Cologne（科隆大学）
HHMI Janelia Research Campus（霍华德·休斯医学研究所珍妮莉亚研究园区）；ETH Zürich（苏黎世联邦理工学院）；MRC Laboratory of Molecular Biology（医学研究理事会分子生物学实验室）
We present a method for neural network interpretability by combining feature attribution with counterfactual explanations to generate attribution maps that highlight the most discriminative features between pairs of classes. We show that this method can be used to quantitatively evaluate the performance of feature attribution methods in an objective manner, thus preventing potential observer bias. We evaluate the proposed method on three diverse datasets, including a challenging artificial dataset and real-world biological data. We show quantitatively and qualitatively that the highlighted fea…
Microsoft Research India（微软印度研究院）；University of Chicago（芝加哥大学）
Feature attributions and counterfactual explanations are popular approaches to explain a ML model. The former assigns an importance score to each input feature, while the latter provides input examples with minimal changes to alter the model's predictions. To unify these approaches, we provide an interpretation based on the actual causality framework and present two key results in terms of their use. First, we present a method to generate feature attribution explanations from a set of counterfactual examples. These feature attributions convey how important a feature is to changing the classifi…
Unifying Image Counterfactuals and Feature Attributions with Latent-Space Adversarial Attacks
Department of Statistics, University of California, Berkeley（加州大学伯克利分校统计学系）；Department of Statistics and Data Science, University of Pennsylvania（宾夕法尼亚大学统计学与数据科学系）
SKEMA Business School（SKEMA商学院）；Millennium Institute for Foundations of Data Research（千禧数据基础研究所）
In this expository article we highlight the relevance of explanations for artificial intelligence, in general, and for the newer developments in {\em explainable AI}, referring to origins and connections of and among different approaches. We describe in simple terms, explanations in data management and machine learning that are based on attribution-scores, and counterfactuals as found in the area of causality. We elaborate on the importance of logical reasoning when dealing with counterfactuals, and their use for score computation.
Feature attributions are a common paradigm for model explanations due to their simplicity in assigning a single numeric score for each input feature to a model. In the actionable recourse setting, wherein the goal of the explanations is to improve outcomes for model consumers, it is often unclear how feature attributions should be correctly used. With this work, we aim to strengthen and clarify the link between actionable recourse and feature attributions. Concretely, we propose a variant of SHAP, Counterfactual SHAP (CF-SHAP), that incorporates counterfactual information to produce a backgrou…
University of California, Irvine（加州大学欧文分校）；Harvard University（哈佛大学）
As machine learning models are increasingly used in critical decision-making settings (e.g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions. Such \textit{explanations} are used to understand and establish trust in models and are vital components in machine learning pipelines. Though explanations are a critical piece in these systems, there is little understanding about how they are vulnerable to manipulation by adversaries. In this paper, we discuss how two broad classes of explanations are vulnerable to manipulation. We demonstrate h…
Friedrich Schiller University Jena（耶拿弗里德里希·席勒大学）；Michael Stifel Center Jena for Data-Driven and Simulation Science（耶拿米夏埃尔·施蒂费尔数据驱动与仿真科学中心）；German Aerospace Center (DLR), Institute for Data Science（德国航空航天中心数据科学研究所）
There are numerous methods for detecting anomalies in time series, but that is only the first step to understanding them. We strive to exceed this by explaining those anomalies. Thus we develop a novel attribution scheme for multivariate time series relying on counterfactual reasoning. We aim to answer the counterfactual question of would the anomalous event have occurred if the subset of the involved variables had been more similarly distributed to the data outside of the anomalous interval. Specifically, we detect anomalous intervals using the Maximally Divergent Interval (MDI) algorithm, re…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of California, San Diego（加州大学圣迭戈分校）
There has been a recent resurgence of interest in explainable artificial intelligence (XAI) that aims to reduce the opaqueness of AI-based decision-making systems, allowing humans to scrutinize and trust them. Prior work in this context has focused on the attribution of responsibility for an algorithm's decisions to its inputs wherein responsibility is typically approached as a purely associational concept. In this paper, we propose a principled causality-based approach for explaining black-box decision-making systems that addresses limitations of existing methods in XAI. At the core of our fr…
We target the problem of detecting Trojans or backdoors in DNNs. Such models behave normally with typical inputs but produce specific incorrect predictions for inputs poisoned with a Trojan trigger. Our approach is based on a novel observation that the trigger behavior depends on a few ghost neurons that activate on trigger pattern and exhibit abnormally higher relative attribution for wrong decisions when activated. Further, these trigger neurons are also active on normal inputs of the target class. Thus, we use counterfactual attributions to localize these ghost neurons from clean inputs and…
School of Computer Science, Peking University, Beijing 100871, China（北京大学计算机科学学院）；Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education（高可信软件技术重点实验室（北京大学））
Towards A Unified Information Bottleneck Framework for Time Series Explanations
Florida International University（佛罗里达国际大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Houston（休斯顿大学）；NEC Labs America（美国 NEC 实验室）；Singapore Management University（新加坡管理大学）
FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation
Quality and Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Deutsche Telekom（德国电信）；Saarland Informatics Campus（萨尔州信息学校区）；Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
Constraint-Anchored Attribution: Feasibility-Certified Counterfactuals and Bonferroni-PAC Sufficient Subsets for Neural CO Policies
Univ. Artois, UR 3926, Laboratoire de G\'enie Informatique et d'Automatique de l'Artois (LGI2A) B\'ethune F-62400 France；Univ. Artois, UR 3926, Laboratoire de G\'enie Informatique et d'Automatique de l'Artois (LGI2A)
Predictive uncertainties in classification tasks are often a consequence of model inadequacy or insufficient training data. In popular applications, such as image processing, we are often required to scrutinise these uncertainties by meaningfully attributing them to input features. This helps to improve interpretability assessments. However, there exist few effective frameworks for this purpose. Vanilla forms of popular methods for the provision of saliency masks, such as SHAP or integrated gradients, adapt poorly to target measures of uncertainty. Thus, state-of-the-art tools instead proceed …
Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals
Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen（机器学习新视角科学卓越中心，图宾根大学）；Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（计算机科学系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Hertie Institute for Artificial Intelligence in Brain Health, University of Tübingen（脑健康人工智能研究所，图宾根大学）；Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern（糖尿病、内分泌学、营养医学和代谢学系，伯恩大学医院，伯恩大学）；Faculty of Health Sciences and Medicine, University of Lucerne（健康科学与医学学院，卢塞恩大学）
Clinical applicability of automated decision support systems depends on a robust, well-understood classification interpretation. Artificial neural networks while achieving class-leading scores fall short in this regard. Therefore, numerous approaches have been proposed that map a salient region of an image to a diagnostic classification. Utilizing heuristic methodology, like blurring and noise, they tend to produce diffuse, sometimes misleading results, hindering their general adoption. In this work we overcome these issues by presenting a model agnostic saliency mapping framework tailored to …
Feature Attribution with Necessity and Sufficiency via Dual-stage Perturbation Test for Causal Explanation
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））；University of Cambridge（剑桥大学）；Shantou University（汕头大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Advancing climate model interpretability: Feature attribution for Arctic melt anomalies
Institute for Harnessing Data and Model Revolution in the Polar Regions（极地地区数据与模型利用研究所）；University of Maryland, Baltimore County（马里兰大学巴尔的摩分校）；National Oceanic and Atmospheric Administration（美国国家海洋和大气管理局）
Towards Desiderata-Driven Design of Visual Counterfactual Explainers
Berlin Institute for the Foundations of Learning（柏林学习与数据基础研究所）；Department of Artificial Intelligence, Korea University, Seoul, Korea（人工智能系，韩国大学，首尔，韩国）；Max-Planck Institute for Informatics, Saarbruecken, Germany（马克斯·普朗克信息研究所，萨尔布吕肯，德国）
Gradients have been used to quantify feature importance in machine learning models. Unfortunately, in nonlinear deep networks, not only individual neurons but also the whole network can saturate, and as a result an important input feature can have a tiny gradient. We study various networks, and observe that this phenomena is indeed widespread, across many inputs. We propose to examine interior gradients, which are gradients of counterfactual inputs constructed by scaling down the original input. We apply our method to the GoogleNet architecture for object recognition in images, as well as a li…
Rice University（莱斯大学）；Texas A&M University（得克萨斯农工大学）；J.P. Morgan AI Research（摩根大通人工智能研究院）
Counterfactual, serving as one emerging type of model explanation, has attracted tons of attentions recently from both industry and academia. Different from the conventional feature-based explanations (e.g., attributions), counterfactuals are a series of hypothetical samples which can flip model decisions with minimal perturbations on queries. Given valid counterfactuals, humans are capable of reasoning under ``what-if'' circumstances, so as to better understand the model decision boundaries. However, releasing counterfactuals could be detrimental, since it may unintentionally leak sensitive i…
With the wide use of deep neural networks (DNN), model interpretability has become a critical concern, since explainable decisions are preferred in high-stake scenarios. Current interpretation techniques mainly focus on the feature attribution perspective, which are limited in indicating why and how particular explanations are related to the prediction. To this end, an intriguing class of explanations, named counterfactuals, has been developed to further explore the "what-if" circumstances for interpretation, and enables the reasoning capability on black-box models. However, generating counter…
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（伊萨尔河畔临床教学中心）
Deep learning models used in medical image analysis are prone to raising reliability concerns due to their black-box nature. To shed light on these black-box models, previous works predominantly focus on identifying the contribution of input features to the diagnosis, i.e., feature attribution. In this work, we explore counterfactual explanations to identify what patterns the models rely on for diagnosis. Specifically, we investigate the effect of changing features within chest X-rays on the classifier's output to understand its decision mechanism. We leverage a StyleGAN-based approach (StyleE…
The paper proposes summarized attribution-based post-hoc explanations for the detection and identification of bias in data. A global explanation is proposed, and a step-by-step framework on how to detect and test bias is introduced. Since removing unwanted bias is often a complicated and tremendous task, it is automatically inserted, instead. Then, the bias is evaluated with the proposed counterfactual approach. The obtained results are validated on a sample skin lesion dataset. Using the proposed method, a number of possible bias causing artifacts are successfully identified and confirmed in …
Exploring Energy Landscapes for Minimal Counterfactual Explanations: Applications in Cybersecurity and Beyond
Netcompany-Intrasoft S.A.（Netcompany-Intrasoft公司）；Hellenic Mediterranean University（希腊地中海大学）；Harokopio University of Athens（雅典哈罗科皮奥大学）；University of Western Macedonia（西马其顿大学）
