Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所）；Institute of Industrial Manufacturing and Management IFF, University of Stuttgart（斯图加特大学工业制造与管理研究所）
Benjamin Fresz, Lena Lörcher, Marco Huber
发表信息：
FAccT'24: The 2024 ACM Conference on Fairness, Accountability, and Transparency, June 2024, Pages 1-19
Decision processes of computer vision models - especially deep neural networks - are opaque in nature, meaning that these decisions cannot be understood by humans. Thus, over the last years, many methods to provide human-understandable explanations have been proposed. For image classification, the most common group are saliency methods, which provide (super-)pixelwise feature attribution scores for input images. But their evaluation still poses a problem, as their results cannot be simply compared to the unknown ground truth. To overcome this, a slew of different proxy metrics have been defined, which are - as the explainability methods themselves - often built on intuition and thus, are possibly unreliable. In this paper, new evaluation metrics for saliency methods are developed and common saliency methods are benchmarked on ImageNet. In addition, a scheme for reliability evaluation of such metrics is proposed that is based on concepts from psychometric testing. The used code can be found at https://github.com/lelo204/ClassificationMetricsForImageExplanations .
University of California, Santa Cruz（加州大学圣克鲁兹分校）；Northwestern University（西北大学）
There is broad agreement that Artificial Intelligence (AI) systems, particularly those using Machine Learning (ML), should be able to "explain" their behavior. Unfortunately, there is little agreement as to what constitutes an "explanation." This has caused a disconnect between the explanations that systems produce in service of explainable Artificial Intelligence (XAI) and those explanations that users and other audiences actually need, which should be defined by the full spectrum of functional roles, audiences, and capabilities for explanation. In this paper, we explore the features of expla…
In healthcare, AI systems support clinicians and patients in diagnosis, treatment, and monitoring, but many systems' poor explainability remains challenging for practical application. Overcoming this barrier is the goal of explainable AI (XAI). However, an explanation can be perceived differently and, thus, not solve the black-box problem for everyone. The domain of Human-Centered AI deals with this problem by adapting AI to users. We present a user-centered persona concept to evaluate XAI and use it to investigate end-users preferences for various explanation styles and contents in a mobile h…
University College Dublin（都柏林大学学院）；Insight Centre for Data Analytics, UCD（UCD洞察数据分析中心）；ML Labs, SFI CRT（SFI CRT机器学习实验室）
Very few eXplainable AI (XAI) studies consider how users understanding of explanations might change depending on whether they know more or less about the to be explained domain (i.e., whether they differ in their expertise). Yet, expertise is a critical facet of most high stakes, human decision making (e.g., understanding how a trainee doctor differs from an experienced consultant). Accordingly, this paper reports a novel, user study (N=96) on how peoples expertise in a domain affects their understanding of post-hoc explanations by example for a deep-learning, black box classifier. The results…
Non-technical end-users are silent and invisible users of the state-of-the-art explainable artificial intelligence (XAI) technologies. Their demands and requirements for AI explainability are not incorporated into the design and evaluation of XAI techniques, which are developed to explain the rationales of AI decisions to end-users and assist their critical decisions. This makes XAI techniques ineffective or even harmful in high-stakes applications, such as healthcare, criminal justice, finance, and autonomous driving systems. To systematically understand end-users' requirements to support the…
Should XAI Nudge Human Decisions with Explanation Biasing?
Tokyo Metropolitan University（东京都立大学）；National Institute of Informatics（信息学研究所）；The Graduate University for Advanced Studies (SOKENDAI)（高级研究大学院大学（SOKENDAI））
School of Engineering and Applied Sciences Washington State University（华盛顿州立大学工程与应用科学学院）；Data Sciences and Analytics Group Pacific Northwest National Laboratory（太平洋西北国家实验室数据科学与分析组）；Cognitive Engineering Center Georgia Institute of Technology（佐治亚理工学院认知工程中心）；Visual Analytics Group Pacific Northwest National Laboratory（太平洋西北国家实验室可视化分析组）
Research into the explanation of machine learning models, i.e., explainable AI (XAI), has seen a commensurate exponential growth alongside deep artificial neural networks throughout the past decade. For historical reasons, explanation and trust have been intertwined. However, the focus on trust is too narrow, and has led the research community astray from tried and true empirical methods that produced more defensible scientific knowledge about people and explanations. To address this, we contribute a practical path forward for researchers in the XAI field. We recommend researchers focus on the…
This paper proposes an alternative approach to the basic taxonomy of explanations produced by explainable artificial intelligence techniques. Methods of Explainable Artificial Intelligence (XAI) were developed to answer the question why a certain prediction or estimation was made, preferably in terms easy to understand by the human agent. XAI taxonomies proposed in the literature mainly concentrate their attention on distinguishing explanations with respect to involving the human agent, which makes it complicated to provide a more mathematical approach to distinguish and compare different expl…
As the societal impact of Deep Neural Networks (DNNs) grows, the goals for advancing DNNs become more complex and diverse, ranging from improving a conventional model accuracy metric to infusing advanced human virtues such as fairness, accountability, transparency (FaccT), and unbiasedness. Recently, techniques in Explainable Artificial Intelligence (XAI) are attracting considerable attention, and have tremendously helped Machine Learning (ML) engineers in understanding AI models. However, at the same time, we started to witness the emerging need beyond XAI among AI communities; based on the i…
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
The boundaries of existing explainable artificial intelligence (XAI) algorithms are confined to problems grounded in technical users' demand for explainability. This research paradigm disproportionately ignores the larger group of non-technical end users, who have a much higher demand for AI explanations in diverse explanation goals, such as making safer and better decisions and improving users' predicted outcomes. Lacking explainability-focused functional support for end users may hinder the safe and accountable use of AI in high-stakes domains, such as healthcare, criminal justice, finance, …
Nowadays, deep neural networks are widely used in mission critical systems such as healthcare, self-driving vehicles, and military which have direct impact on human lives. However, the black-box nature of deep neural networks challenges its use in mission critical applications, raising ethical and judicial concerns inducing lack of trust. Explainable Artificial Intelligence (XAI) is a field of Artificial Intelligence (AI) that promotes a set of tools, techniques, and algorithms that can generate high-quality interpretable, intuitive, human-understandable explanations of AI decisions. In additi…
Although standard Machine Learning models are optimized for making predictions about observations, more and more they are used for making predictions about the results of actions. An important goal of Explainable Artificial Intelligence (XAI) is to compensate for this mismatch by offering explanations about the predictions of an ML-model which ensure that they are reliably action-guiding. As action-guiding explanations are causal explanations, the literature on this topic is starting to embrace insights from the literature on causal models. Here I take a step further down this path by formally…
Despite the proliferation of explainable AI (XAI) methods, little is understood about end-users' explainability needs and behaviors around XAI explanations. To address this gap and contribute to understanding how explainability can support human-AI interaction, we conducted a mixed-methods study with 20 end-users of a real-world AI application, the Merlin bird identification app, and inquired about their XAI needs, uses, and perceptions. We found that participants desire practically useful information that can improve their collaboration with the AI, more so than technical system details. Rela…
XAI with natural language processing aims to produce human-readable explanations as evidence for AI decision-making, which addresses explainability and transparency. However, from an HCI perspective, the current approaches only focus on delivering a single explanation, which fails to account for the diversity of human thoughts and experiences in language. This paper thus addresses this gap, by proposing a generative XAI framework, INTERACTION (explaIn aNd predicT thEn queRy with contextuAl CondiTional varIational autO-eNcoder). Our novel framework presents explanation in two steps: (step one) …
Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI
Insight SFI Centre for Data Analytics, School of Computer Science, University College Dublin（都柏林大学学院计算机科学学院Insight SFI数据分析中心）；VistaMilk SFI Research Centre, School of Computer Science, University College Dublin（都柏林大学学院计算机科学学院VistaMilk SFI研究中心）；School of Psychology and Institute of Neuroscience, Trinity College Dublin, University of Dublin（都柏林圣三一学院心理学院和神经科学研究所）
Beuth University of Applied Sciences（柏林应用科学大学（Beuth））；Einstein Center Digital Future（爱因斯坦数字未来中心）
A central goal of explainable artificial intelligence (XAI) is to improve the trust relationship in human-AI interaction. One assumption underlying research in transparent AI systems is that explanations help to better assess predictions of machine learning (ML) models, for instance by enabling humans to identify wrong predictions more efficiently. Recent empirical evidence however shows that explanations can have the opposite effect: When presenting explanations of ML predictions humans often tend to trust ML predictions even when these are wrong. Experimental evidence suggests that this effe…
Ericsson Research（爱立信研究院）；Imperial College London（帝国理工学院）；Università degli Studi di Brescia（布雷西亚大学）
Explainable AI (XAI) has been investigated for decades and, together with AI itself, has witnessed unprecedented growth in recent years. Among various approaches to XAI, argumentative models have been advocated in both the AI and social science literature, as their dialectical nature appears to match some basic desirable features of the explanation activity. In this survey we overview XAI approaches built using methods from the field of computational argumentation, leveraging its wide array of reasoning abstractions and explanation delivery methods. We overview the literature focusing on diffe…
Explainable artificial intelligence (XAI) methods are currently evaluated with approaches mostly originated in interpretable machine learning (IML) research that focus on understanding models such as comparison against existing attribution approaches, sensitivity analyses, gold set of features, axioms, or through demonstration of images. There are problems with these methods such as that they do not indicate where current XAI approaches fail to guide investigations towards consistent progress of the field. They do not measure accuracy in support of accountable decisions, and it is practically …
The main objective of eXplainable Artificial Intelligence (XAI) is to provide effective explanations for black-box classifiers. The existing literature lists many desirable properties for explanations to be useful, but there is no consensus on how to quantitatively evaluate explanations in practice. Moreover, explanations are typically used only to inspect black-box models, and the proactive use of explanations as a decision support is generally overlooked. Among the many approaches to XAI, a widely adopted paradigm is Local Linear Explanations - with LIME and SHAP emerging as state-of-the-art…
Pennsylvania State University（宾夕法尼亚州立大学）；College of Information Sciences and Technology（信息科学与技术学院）
Artificial intelligence (AI) has become prevalent in our everyday technologies and impacts both individuals and communities. The explainable AI (XAI) scholarship has explored the philosophical nature of explanation and technical explanations, which are usually driven by experts in lab settings and can be challenging for laypersons to understand. In addition, existing XAI research tends to focus on the individual level. Little is known about how people understand and explain AI-led decisions in the community context. Drawing from XAI and activity theory, a foundational HCI theory, we theorize h…
Explainable Artificial Intelligence (XAI) has in recent years become a well-suited framework to generate human understandable explanations of "black-box" models. In this paper, a novel XAI visual explanation algorithm known as the Similarity Difference and Uniqueness (SIDU) method that can effectively localize entire object regions responsible for prediction is presented in full detail. The SIDU algorithm robustness and effectiveness is analyzed through various computational and human subject experiments. In particular, the SIDU algorithm is assessed using three different types of evaluations …
In recent years, there has been an explosion of AI research on counterfactual explanations as a solution to the problem of eXplainable AI (XAI). These explanations seem to offer technical, psychological and legal benefits over other explanation techniques. We survey 100 distinct counterfactual explanation methods reported in the literature. This survey addresses the extent to which these methods have been adequately evaluated, both psychologically and computationally, and quantifies the shortfalls occurring. For instance, only 21% of these methods have been user tested. Five key deficits in th…
EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of…
University College Dublin（都柏林大学学院）；Insight Centre for Data Analytics（数据分析洞察中心）；VistaMilk SFI Research Centre（VistaMilk SFI研究中心）
This paper surveys an approach to the XAI problem, using post-hoc explanation by example, that hinges on twinning Artificial Neural Networks (ANNs) with Case-Based Reasoning (CBR) systems, so-called ANN-CBR twins. A systematic survey of 1100+ papers was carried out to identify the fragmented literature on this topic and to trace it influence through to more recent work involving Deep Neural Networks (DNNs). The paper argues that this twin-system approach, especially using ANN-CBR twins, presents one possible coherent, generic solution to the XAI problem (and, indeed, XCBR problem). The paper c…
Seoul National University（首尔大学）；Carnegie Mellon University（卡内基梅隆大学）
Explainable AI (XAI) is a promising means of supporting human-AI collaborations for high-stakes visual detection tasks, such as damage detection tasks from satellite imageries, as fully-automated approaches are unlikely to be perfectly safe and reliable. However, most existing XAI techniques are not informed by the understandings of task-specific needs of humans for explanations. Thus, we took a first step toward understanding what forms of XAI humans require in damage detection tasks. We conducted an online crowdsourced study to understand how people explain their own assessments, when evalua…
Explainable AI-Based Interface System for Weather Forecasting Model
Korea Advanced Institute of Science and Technology(KAIST)（韩国科学技术院(KAIST)）；SI-Analytics；National Institute of Meteorological Sciences(NIMS)（气象厅气象研究所(NIMS)）；INEEJI
On the Hardness of Computing Counterfactual and Semifactual Explanations in XAI
Faculty of Technology（技术学院）；Bielefeld University（比勒菲尔德大学）；Department of Business Development and Technology（商业发展与技术系）；Aarhus University（哥本哈根大学）；Department of Business Decisions and Analytics（商业决策与分析系）；University of Vienna（维也纳大学）
For Better or Worse: The Impact of Counterfactual Explanations' Directionality on User Behavior in xAI
Bielefeld University（比勒费尔德大学）；Research Institute for Cognition and Robotics (CoR-Lab)（认知与机器人研究所（CoR-Lab））；Machine Learning Group, Faculty of Technology（技术学院机器学习组）
