There have been several research works proposing new Explainable AI (XAI) methods designed to generate model explanations having specific properties, or desiderata, such as fidelity, robustness, or human-interpretability. However, explanations are seldom evaluated based on their true practical impact on decision-making tasks. Without that assessment, explanations might be chosen that, in fact, hurt the overall performance of the combined system of ML model + end-users. This study aims to bridge this gap by proposing XAI Test, an application-grounded evaluation methodology tailored to isolate the impact of providing the end-user with different levels of information. We conducted an experiment following XAI Test to evaluate three popular post-hoc explanation methods -- LIME, SHAP, and TreeInterpreter -- on a real-world fraud detection task, with real data, a deployed ML model, and fraud analysts. During the experiment, we gradually increased the information provided to the fraud analysts in three stages: Data Only, i.e., just transaction data without access to model score nor explanations, Data + ML Model Score, and Data + ML Model Score + Explanations. Using strong statistical analysis, we show that, in general, these popular explainers have a worse impact than desired. Some of the conclusion highlights include: i) showing Data Only results in the highest decision accuracy and the slowest decision time among all variants tested, ii) all the explainers improve accuracy over the Data + ML Model Score variant but still result in lower accuracy when compared with Data Only; iii) LIME was the least preferred by users, probably due to its substantially lower variability of explanations from case to case.
University of Toronto（多伦多大学）；University of Cambridge（剑桥大学）
Recent work has suggested post-hoc explainers might be ineffective for detecting spurious correlations in Deep Neural Networks (DNNs). However, we show there are serious weaknesses with the existing evaluation frameworks for this setting. Previously proposed metrics are extremely difficult to interpret and are not directly comparable between explainer methods. To alleviate these constraints, we propose a new evaluation methodology, Explainer Divergence Scores (EDS), grounded in an information theory approach to evaluate explainers. EDS is easy to interpret and naturally comparable across expla…
Many applications of data-driven models demand transparency of decisions, especially in health care, criminal justice, and other high-stakes environments. Modern trends in machine learning research have led to algorithms that are increasingly intricate to the degree that they are considered to be black boxes. In an effort to reduce the opacity of decisions, methods have been proposed to construe the inner workings of such models in a human-comprehensible manner. These post hoc techniques are described as being universal explainers - capable of faithfully augmenting decisions with algorithmic i…
As neural networks increasingly make critical decisions in high-stakes settings, monitoring and explaining their behavior in an understandable and trustworthy manner is a necessity. One commonly used type of explainer is post hoc feature attribution, a family of methods for giving each feature in an input a score corresponding to its influence on a model's output. A major limitation of this family of explainers in practice is that they can disagree on which features are more important than others. Our contribution in this paper is a method of training models with this disagreement problem in m…
Prototypical Self-Explainable Models Without Re-training
UiT The Arctic University of Norway（挪威北极大学）；Physikalisch-Technische Bundesanstalt（德国联邦物理技术研究院）；Leibniz Institute for Agriculture and Bioeconomy（莱布尼茨农业与生物经济研究所）；University of Potsdam（波茨坦大学）
Towards Bridging the FL Performance-Explainability Trade-Off: A Trustworthy 6G RAN Slicing Use-Case
Iquadrat Informatica（伊夸德拉特信息公司）；Universitat Politècnica de Catalunya (UPC)（加泰罗尼亚理工大学）；i2CAT Foundation（i2CAT基金会）；University of Patras（帕特雷大学）；ISI/ATHENA（雅典娜创新基金会信息研究所）
A novel post-hoc explanation comparison metric and applications
Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；University of California at Santa Cruz（加州大学圣克鲁兹分校）
University of Kentucky（肯塔基大学）；University of Missouri（密苏里大学）
There have been several post-hoc explanation approaches developed to explain pre-trained black-box neural networks. However, there is still a gap in research efforts toward designing neural networks that are inherently explainable. In this paper, we utilize a recently proposed instance-wise post-hoc causal explanation method to make an existing transformer architecture inherently explainable. Once trained, our model provides an explanation in the form of top-$k$ regions in the input space of the given instance contributing to its decision. We evaluate our method on binary classification tasks …
A critical problem in the field of post hoc explainability is the lack of a common foundational goal among methods. For example, some methods are motivated by function approximation, some by game theoretic notions, and some by obtaining clean visualizations. This fragmentation of goals causes not only an inconsistent conceptual understanding of explanations but also the practical challenge of not knowing which method to use when. In this work, we begin to address these challenges by unifying eight popular post hoc explanation methods (LIME, C-LIME, KernelSHAP, Occlusion, Vanilla Gradients, Gra…
Discovering Efficient and Explainable Communication Topologies for LLM-based Multi-Agent Systems via Causal Inference
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）
Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam（计算语言学与文本挖掘实验室，荷兰自由大学）；TNO–The Netherlands Organization for Applied Scientific Research, The Hague（荷兰应用科学研究院（TNO），海牙）；Leiden University Centre for Linguistics (LUCL), Leiden University, Leiden（莱顿大学语言中心（LUCL），莱顿大学，莱顿）
A Two-Stage LLM Framework for Accessible and Verified XAI Explanations
Department of Computer Engineering and Informatics, University of Patras（帕特拉大学计算机工程与信息学系）；Department of Informatics and Telecommunications, University of Ioannina（伊奥安尼纳大学信息与电信系）；Industrial Systems Institute, Athena Research Center（雅典研究中心工业系统研究所）；Archimedes Unit, Athena Research Center（雅典研究中心阿基米德单位）
Explaining Time Series Classifiers with PHAR: Rule Extraction and Fusion from Post-hoc Attributions
Jagiellonian Human-Centered AI Lab, Mark Kac Center for Complex Systems Research（杰克利昂人类中心人工智能实验室、马克·卡复杂系统研究中心）；Institute of Applied Computer Science, Jagiellonian University（应用计算机科学研究所、杰克利昂大学）
University of Toronto（多伦多大学）；LG AI Research（LG人工智能研究院）
Recent years have seen the introduction of a range of methods for post-hoc explainability of image classifier predictions. However, these post-hoc explanations may not always be faithful to classifier predictions, which poses a significant challenge when attempting to debug models based on such explanations. To this end, we seek a methodology that can improve the faithfulness of an explanation method with respect to model predictions which does not require ground truth explanations. We achieve this through a novel explanation-driven data augmentation (EDDA) technique that augments the training…
Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；University of Trento（特伦托大学）；Free University of Bozen Bolzano（博尔扎诺自由大学）
Recent papers have introduced a novel approach to explain why a Predictive Process Monitoring (PPM) model for outcome-oriented predictions provides wrong predictions. Moreover, they have shown how to exploit the explanations, obtained using state-of-the art post-hoc explainers, to identify the most common features that induce a predictor to make mistakes in a semi-automated way, and, in turn, to reduce the impact of those features and increase the accuracy of the predictive model. This work starts from the assumption that frequent control flow patterns in event logs may represent important fea…
