We present a focused analysis of user studies in explainable artificial intelligence (XAI) entailing qualitative investigation. We draw on social science corpora to suggest ways for improving the rigor of studies where XAI researchers use observations, interviews, focus groups, and/or questionnaires to capture qualitative data. We contextualize the presentation of the XAI papers included in our analysis according to the components of rigor described in the qualitative research literature: 1) underlying theories or frameworks, 2) methodological approaches, 3) data collection methods, and 4) data analysis processes. The results of our analysis support calls from others in the XAI community advocating for collaboration with experts from social disciplines to bolster rigor and effectiveness in user studies.
A Differentiable Atari VCS:A Complex, Fully Known Ground Truth for Explainable AI
Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，弗里德里希-亚历山大-埃森纳赫大学）；Mannheim Center for Neuromodulation and Neuroprosthetics, Heidelberg University（曼海姆神经调制与神经假体中心，海德堡大学）
University of Stuttgart（斯图加特大学）；Fraunhofer Institute for Industrial Engineering IAO（弗劳恩霍夫工业工程研究所）
With recent progress in the field of Explainable Artificial Intelligence (XAI) and increasing use in practice, the need for an evaluation of different XAI methods and their explanation quality in practical usage scenarios arises. For this purpose, we present VitrAI, which is a web-based service with the goal of uniformly demonstrating four different XAI algorithms in the context of three real life scenarios and evaluating their performance and comprehensibility for humans. This work reveals practical obstacles when adopting XAI methods and gives qualitative estimates on how well different appr…
Explainable AI-Based Interface System for Weather Forecasting Model
Korea Advanced Institute of Science and Technology(KAIST)（韩国科学技术院(KAIST)）；SI-Analytics；National Institute of Meteorological Sciences(NIMS)（气象厅气象研究所(NIMS)）；INEEJI
A new brand of technical artificial intelligence ( Explainable AI ) research has focused on trying to open up the 'black box' and provide some explainability. This paper presents a novel visual explanation method for deep learning networks in the form of a saliency map that can effectively localize entire object regions. In contrast to the current state-of-the art methods, the proposed method shows quite promising visual explanations that can gain greater trust of human expert. Both quantitative and qualitative evaluations are carried out on both general and clinical data sets to confirm the e…
Recent advancements in AI have coincided with ever-increasing efforts in the research community to investigate, classify and evaluate various methods aimed at making AI models explainable. However, most of existing attempts present a method-centric view of eXplainable AI (XAI) which is typically meaningful only for domain experts. There is an apparent lack of a robust qualitative and quantitative performance framework that evaluates the suitability of explanations for different types of users. We survey relevant efforts, and then, propose a unified, inclusive and user-centred taxonomy for XAI …
Friedrich-Alexander University Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学）
For many applications, utilizing DNNs (Deep Neural Networks) requires their implementation on a target architecture in an optimized manner concerning energy consumption, memory requirement, throughput, etc. DNN compression is used to reduce the memory footprint and complexity of a DNN before its deployment on hardware. Recent efforts to understand and explain AI (Artificial Intelligence) methods have led to a new research area, termed as explainable AI. Explainable AI methods allow us to understand better the inner working of DNNs, such as the importance of different neurons and features. The …
The ability to explain the prediction of deep learning models to end-users is an important feature to leverage the power of artificial intelligence (AI) for the medical decision-making process, which is usually considered non-transparent and challenging to comprehend. In this paper, we apply state-of-the-art eXplainable artificial intelligence (XAI) methods to explain the prediction of the black-box AI models in the thyroid nodule diagnosis application. We propose new statistic-based XAI methods, namely Kernel Density Estimation and Density map, to explain the case of no nodule detected. XAI m…
There has been a recent surge in the field of Explainable AI (XAI) which tackles the problem of providing insights into the behavior of black-box machine learning models. Within this field, \textit{feature attribution} encompasses methods which assign relevance scores to input features and visualize them as a heatmap. Designing flexible accelerators for multiple such algorithms is challenging since the hardware mapping of these algorithms has not been studied yet. In this work, we first analyze the dataflow of gradient backpropagation based feature attribution algorithms to determine the resou…
Prototypical Self-Explainable Models Without Re-training
UiT The Arctic University of Norway（挪威北极大学）；Physikalisch-Technische Bundesanstalt（德国联邦物理技术研究院）；Leibniz Institute for Agriculture and Bioeconomy（莱布尼茨农业与生物经济研究所）；University of Potsdam（波茨坦大学）
Evaluating RE Practices for Explainability: Synthesizing Insights from Daimler Truck into an Explainable RE Framework Proposal
TUM School of CIT, Technical University of Munich（慕尼黑技术大学 CIT 学院）；Institute of Software Engineering, University of Stuttgart（斯图加特大学软件工程研究所）；Department of Computer Science, Vrije Universiteit Amsterdam（阿姆斯特丹自由大学计算机科学系）
In this paper, we present a novel method to compute decision rules to build a more accurate interpretable machine learning model, denoted as ExMo. The ExMo interpretable machine learning model consists of a list of IF...THEN... statements with a decision rule in the condition. This way, ExMo naturally provides an explanation for a prediction using the decision rule that was triggered. ExMo uses a new approach to extract decision rules from the training data using term frequency-inverse document frequency (TF-IDF) features. With TF-IDF, decision rules with feature values that are more relevant …
Idiap Research Institute（Idiap研究所）；University of Manchester（曼彻斯特大学）
A fundamental research goal for Explainable AI (XAI) is to build models that are capable of reasoning through the generation of natural language explanations. However, the methodologies to design and evaluate explanation-based inference models are still poorly informed by theoretical accounts on the nature of explanation. As an attempt to provide an epistemologically grounded characterisation for XAI, this paper focuses on the scientific domain, aiming to bridge the gap between theory and practice on the notion of a scientific explanation. Specifically, the paper combines a detailed survey of …
Explainable AI for Enhancing Efficiency of DL-based Channel Estimation
SogetiLabs Research and Innovation (part of Capgemini)（SogetiLabs研究与创新中心（凯捷集团旗下））；IMT Nord Europe（北欧电信与信息技术学院）；National Center for Remote Sensing - CNRS（法国国家科学研究中心国家遥感中心）
Sparse, Efficient and Explainable Data Attribution with DualXDA
Department of Artificial Intelligence Fraunhofer Heinrich Hertz Institute（人工智能系 汉诺威弗劳恩霍夫研究所）；Department of Electrical Engineering and Computer Science Technische Universität Berlin（电气工程与计算机科学系 柏林技术大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Centre of eXplainable Artificial Intelligence Technological University Dublin（可解释人工智能中心 立陶宛技术大学）
National Heart and Lung Institute, Imperial College London（帝国理工学院国家心肺研究所）；Royal Brompton Hospital（皇家布朗普顿医院）；Imperial Institute of Advanced Technology（帝国理工高等研究院）；Hangzhou Ocean’s Smart Boya Co., Ltd（杭州海洋智能博雅有限公司）；University of California, San Diego（加州大学圣地亚哥分校）；Shenzhen Second People’s Hospital（深圳市第二人民医院）
Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems' black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms can not manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI bec…
In this work, we present a novel human-in-the-loop framework to help the human user understand the decision making process that involves choosing preferred options. We focus on qualitative preference models over alternatives from combinatorial domains. This framework is interactive: the user provides her behavioral data to the framework, and the framework explains the learned model to the user. It is iterative: the framework collects feedback on the learned model from the user and tries to improve it accordingly till the user terminates the iteration. In order to communicate the learned prefer…
Law codes and regulations help organise societies for centuries, and as AI systems gain more autonomy, we question how human-agent systems can operate as peers under the same norms, especially when resources are contended. We posit that agents must be accountable and explainable by referring to which rules justify their decisions. The need for explanations is associated with user acceptance and trust. This paper's contribution is twofold: i) we propose an argumentation-based human-agent architecture to map human regulations into a culture for artificial agents with explainable behaviour. Our a…
IBM Research（IBM研究院）；Federal University of Rio Grande do Sul（南里奥格兰德联邦大学）；Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）
The presumed data owners' right to explanations brought about by the General Data Protection Regulation in Europe has shed light on the social challenges of explainable artificial intelligence (XAI). In this paper, we present a case study with Deep Learning (DL) experts from a research and development laboratory focused on the delivery of industrial-strength AI technologies. Our aim was to investigate the social meaning (i.e. meaning to others) that DL experts assign to what they do, given a richly contextualized and familiar domain of application. Using qualitative research techniques to coll…
Deep learning models for learning analytics have become increasingly popular over the last few years; however, these approaches are still not widely adopted in real-world settings, likely due to a lack of trust and transparency. In this paper, we tackle this issue by implementing explainable AI methods for black-box neural networks. This work focuses on the context of online and blended learning and the use case of student success prediction models. We use a pairwise study design, enabling us to investigate controlled differences between pairs of courses. Our analyses cover five course pairs t…
Recent works have recognized the need for human-centered perspectives when designing and evaluating human-AI interactions and explainable AI methods. Yet, current approaches fall short at intercepting and managing unexpected user behavior resulting from the interaction with AI systems and explainability methods of different stake-holder groups. In this work, we explore the use of AI and explainability methods in the insurance domain. In an qualitative case study with participants with different roles and professional backgrounds, we show that AI and explainability methods are used in creative …
