The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practices of more than 300 papers published in the last 7 years at major AI and ML conferences that introduce an XAI method. We find that 1 in 3 papers evaluate exclusively with anecdotal evidence, and 1 in 5 papers evaluate with users. This survey also contributes to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. Our systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark and compare new and existing XAI methods. The Co-12 categorization scheme and our identified evaluation methods open up opportunities to include quantitative metrics as optimization criteria during model training in order to optimize for accuracy and interpretability simultaneously.
Explainable AI has attracted much research attention in recent years with feature attribution algorithms, which compute "feature importance" in predictions, becoming increasingly popular. However, there is little analysis of the validity of these algorithms as there is no "ground truth" in the existing datasets to validate their correctness. In this work, we develop a method to quantitatively evaluate the correctness of XAI algorithms by creating datasets with known explanation ground truth. To this end, we focus on the binary classification problems. String datasets are constructed using form…
Many ML models are opaque to humans, producing decisions too complex for humans to easily understand. In response, explainable artificial intelligence (XAI) tools that analyze the inner workings of a model have been created. Despite these tools' strength in translating model behavior, critiques have raised concerns about the impact of XAI tools as a tool for `fairwashing` by misleading users into trusting biased or incorrect models. In this paper, we created a framework for evaluating explainable AI tools with respect to their capabilities for detecting and addressing issues of bias and fairne…
Predictive process analytics focuses on predicting the future states of running instances of a business process. While advanced machine learning techniques have been used to increase accuracy of predictions, the resulting predictive models lack transparency. Current explainable machine learning methods, such as LIME and SHAP, can be used to interpret black box models. However, it is unclear how fit for purpose these methods are in explaining process predictive models. In this paper, we draw on evaluation measures used in the field of explainable AI and propose functionally-grounded evaluation …
Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of act…
A Unified Framework for Evaluating and Enhancing the Transparency of Explainable AI Methods via Perturbation-Gradient Consensus Attribution
Department of Computer Science, American International University-Bangladesh（美国国际大学-孟加拉国计算机科学系）；Thomas Lord Department of Computer Science, Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院托马斯·洛德计算机科学系）；Department of Computer Science and Engineering, Techno International New Town（Techno International New Town计算机科学与工程系）
Faithfulness Without Anchors: A Position Paper on the Structural Misalignments of Explainable AI Faithfulness Metrics for Feature Importance
TU Delft（代尔夫特理工大学）；XITASO GmbH（XITASO有限公司）；University College Cork（科克大学学院）；RPTU University Kaiserslautern-Landau（凯撒斯劳滕-兰道工业大学）；DFKI（德国人工智能研究中心）；TU Dresden（德累斯顿理工大学）
Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposal
TUM School of CIT, Technical University of Munich（慕尼黑技术大学 CIT 学院）；Institute of Software Engineering, University of Stuttgart（斯图加特大学软件工程研究所）；Department of Computer Science, Vrije Universiteit Amsterdam（阿姆斯特丹自由大学计算机科学系）
Evaluating Single and Multi-Omics Based Explainable Artificial Intelligence (MOXAI) for Molecular Subclass Classification of Adult-Type Diffuse Gliomas
Explainable Artificial Intelligence Model for Evaluating Shear Strength Parameters of Municipal Solid Waste Across Diverse Compositional Profiles
The Joint Graduate School of Energy and Environment, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区能源与环境联合研究生院）；AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区工程学院土木工程系人工智能研究组）；Center of Excellence on Energy Technology and Environment (CEE), PERDO, Ministry of Higher Education, Science, Research and Innovation (MHESI)（高等教育、科学、研究与创新部PERDO能源技术与环境卓越中心）；Department of Environmental Engineering, Faculty of Engineering, Kasetsart University（农业大学工程学院环境工程系）
Algorithmic approaches to interpreting machine learning models have proliferated in recent years. We carry out human subject tests that are the first of their kind to isolate the effect of algorithmic explanations on a key aspect of model interpretability, simulatability, while avoiding important confounding experimental factors. A model is simulatable when a person can predict its behavior on new inputs. Through two kinds of simulation tests involving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Com…
Institute for Human and Machine Cognition（人机认知研究所）；Michigan Technological University（密歇根理工大学）；Macrocognition, LLC（宏认知有限责任公司）
The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We discuss specific methods for evaluating: (1) the goodness of explanations, (2) whether users are satisfied by explanations, (3) how well users understand the AI systems, (4) how curiosity motivates the search for explanations, (5) whether the user's tr…
University of Stuttgart（斯图加特大学）；Fraunhofer Institute for Industrial Engineering IAO（弗劳恩霍夫工业工程研究所）
With recent progress in the field of Explainable Artificial Intelligence (XAI) and increasing use in practice, the need for an evaluation of different XAI methods and their explanation quality in practical usage scenarios arises. For this purpose, we present VitrAI, which is a web-based service with the goal of uniformly demonstrating four different XAI algorithms in the context of three real life scenarios and evaluating their performance and comprehensibility for humans. This work reveals practical obstacles when adopting XAI methods and gives qualitative estimates on how well different appr…
The increasingly widespread application of AI models motivates increased demand for explanations from a variety of stakeholders. However, this demand is ambiguous because there are many types of 'explanation' with different evaluative criteria. In the spirit of pluralism, I chart a taxonomy of types of explanation and the associated XAI methods that can address them. When we look to expose the inner mechanisms of AI models, we develop Diagnostic-explanations. When we seek to render model output understandable, we produce Explication-explanations. When we wish to form stable generalizations of …
Machine learning models need to provide contrastive explanations, since people often seek to understand why a puzzling prediction occurred instead of some expected outcome. Current contrastive explanations are rudimentary comparisons between examples or raw features, which remain difficult to interpret, since they lack semantic meaning. We argue that explanations must be more relatable to other concepts, hypotheticals, and associations. Inspired by the perceptual process from cognitive psychology, we propose the XAI Perceptual Processing Framework and RexNet model for relatable explainable AI …
Seoul National University（首尔大学）；Carnegie Mellon University（卡内基梅隆大学）
Explainable AI (XAI) is a promising means of supporting human-AI collaborations for high-stakes visual detection tasks, such as damage detection tasks from satellite imageries, as fully-automated approaches are unlikely to be perfectly safe and reliable. However, most existing XAI techniques are not informed by the understandings of task-specific needs of humans for explanations. Thus, we took a first step toward understanding what forms of XAI humans require in damage detection tasks. We conducted an online crowdsourced study to understand how people explain their own assessments, when evalua…
X-SHIELD: Regularization for eXplainable Artificial Intelligence
Department of Computer Science and Artificial Intelligence（计算机科学与人工智能系）；Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI)（数据科学与计算智能安达卢西亚研究机构（DaSCI））；University of Granada（格拉纳达大学）
Research in artificial intelligence (AI)-assisted decision-making is experiencing tremendous growth with a constantly rising number of studies evaluating the effect of AI with and without techniques from the field of explainable AI (XAI) on human decision-making performance. However, as tasks and experimental setups vary due to different objectives, some studies report improved user decision-making performance through XAI, while others report only negligible effects. Therefore, in this article, we present an initial synthesis of existing research on XAI studies using a statistical meta-analysi…
On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy
Dresden University of Technology（德累斯顿技术大学）；Charité Berlin（柏林夏里特医学院）；Department Speculative Transformation（推测转化部门）；Chair of Engineering Psychology and Applied Cognitive Research（工程心理学与应用认知研究教授职位）
Saarland University（萨尔大学）；Technical University Dortmund（多特蒙德工业大学）；Institute of Intelligent Software（智能软件研究所）
Previous research in Explainable Artificial Intelligence (XAI) suggests that a main aim of explainability approaches is to satisfy specific interests, goals, expectations, needs, and demands regarding artificial systems (we call these stakeholders' desiderata) in a variety of contexts. However, the literature on XAI is vast, spreads out across multiple largely disconnected disciplines, and it often remains unclear how explainability approaches are supposed to achieve the goal of satisfying stakeholders' desiderata. This paper discusses the main classes of stakeholders calling for explainabilit…
Recent advancements in AI have coincided with ever-increasing efforts in the research community to investigate, classify and evaluate various methods aimed at making AI models explainable. However, most of existing attempts present a method-centric view of eXplainable AI (XAI) which is typically meaningful only for domain experts. There is an apparent lack of a robust qualitative and quantitative performance framework that evaluates the suitability of explanations for different types of users. We survey relevant efforts, and then, propose a unified, inclusive and user-centred taxonomy for XAI …
Breast Cancer Diagnosis: A Comprehensive Exploration of Explainable Artificial Intelligence (XAI) Techniques
Canadian Institute of Cybersecurity (CIC)（加拿大网络安全研究所）；University of New Brunswick（新不伦瑞克大学）；Dipartimento di Informatica, Università di Verona（威尼斯大学信息学院）；Università di Verona（威尼斯大学）；Department of Computer Science, School of Mathematics and Computer Science, Institute of Business Administration（计算机科学系，数学与计算机科学学院，商学院）；LIRIS, CNRS, Université Claude Bernard Lyon 1（LIRIS，国家科学研究中心，克莱尔伯恩大学 Lyon 1 分校）；LHC, UMR5516, Université de Saint-Etienne（LHC，UMR5516，圣艾蒂安大学）
IDIADA Fahrzeugtechnik GmbH（伊迪亚达汽车技术有限公司）；University of Munich (LMU)（慕尼黑大学）；Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；ZBW - Leibniz Centre for Economics（莱布尼茨经济信息中心）；Kiel University（基尔大学）
In recent years, Explainable AI (xAI) attracted a lot of attention as various countries turned explanations into a legal right. xAI allows for improving models beyond the accuracy metric by, e.g., debugging the learned pattern and demystifying the AI's behavior. The widespread use of xAI brought new challenges. On the one hand, the number of published xAI algorithms underwent a boom, and it became difficult for practitioners to select the right tool. On the other hand, some experiments did highlight how easy data scientists could misuse xAI algorithms and misinterpret their results. To tackle …
Institute for Multimedia and Interactive Systems（多媒体与交互系统研究所）；University of Lübeck（吕贝克大学）
To achieve optimal human-system integration in the context of user-AI interaction it is important that users develop a valid representation of how AI works. In most of the everyday interaction with technical systems users construct mental models (i.e., an abstraction of the anticipated mechanisms a system uses to perform a given task). If no explicit explanations are provided by a system (e.g. by a self-explaining AI) or other sources (e.g. an instructor), the mental model is typically formed based on experiences, i.e. the observations of the user during the interaction. The congruence of this…
This work documents a pilot user study evaluating the effectiveness of contrastive, causal and example explanations in supporting human understanding of AI in a hypothetical commonplace human robot interaction HRI scenario. In doing so, this work situates explainable AI XAI in the context of the social sciences and suggests that HRI explanations are improved when informed by the social sciences.
Universidad Autonoma de Madrid（马德里自治大学）；Laboratoire des Sciences du Numérique de Nantes（南特数字科学实验室）；National Institute of Informatics（国立信息学研究所）
Machine learning methods are growing in relevance for biometrics and personal information processing in domains such as forensics, e-health, recruitment, and e-learning. In these domains, white-box (human-readable) explanations of systems built on machine learning methods can become crucial. Inductive Logic Programming (ILP) is a subfield of symbolic AI aimed to automatically learn declarative theories about the process of data. Learning from Interpretation Transition (LFIT) is an ILP technique that can learn a propositional logic theory equivalent to a given black-box system (under certain co…
