In this work, we report the practical and theoretical aspects of Explainable AI (XAI) identified in some fundamental literature. Although there is a vast body of work on representing the XAI backgrounds, most of the corpuses pinpoint a discrete direction of thoughts. Providing insights into literature in practice and theory concurrently is still a gap in this field. This is important as such connection facilitates a learning process for the early stage XAI researchers and give a bright stand for the experienced XAI scholars. Respectively, we first focus on the categories of black-box explanation and give a practical example. Later, we discuss how theoretically explanation has been grounded in the body of multidisciplinary fields. Finally, some directions of future works are presented.
Center for Artificial Intelligence Research（人工智能研究中心）；University of Agder（阿格德大学）
The past decade has seen significant progress in artificial intelligence (AI), which has resulted in algorithms being adopted for resolving a variety of problems. However, this success has been met by increasing model complexity and employing black-box AI models that lack transparency. In response to this need, Explainable AI (XAI) has been proposed to make AI more transparent and thus advance the adoption of AI in critical domains. Although there are several reviews of XAI topics in the literature that identified challenges and potential research directions in XAI, these challenges and resear…
Royal Bank of Canada（加拿大皇家银行）；Innopolis University（Innopolis 大学）；University of Manitoba（曼尼托巴大学）
The remarkable advancements in Deep Learning (DL) algorithms have fueled enthusiasm for using Artificial Intelligence (AI) technologies in almost every domain; however, the opaqueness of these algorithms put a question mark on their applications in safety-critical systems. In this regard, the `explainability' dimension is not only essential to both explain the inner workings of black-box algorithms, but it also adds accountability and transparency dimensions that are of prime importance for regulators, consumers, and service providers. eXplainable Artificial Intelligence (XAI) is the set of te…
University of Stuttgart（斯图加特大学）；Fraunhofer Institute for Industrial Engineering IAO（弗劳恩霍夫工业工程研究所）
With recent progress in the field of Explainable Artificial Intelligence (XAI) and increasing use in practice, the need for an evaluation of different XAI methods and their explanation quality in practical usage scenarios arises. For this purpose, we present VitrAI, which is a web-based service with the goal of uniformly demonstrating four different XAI algorithms in the context of three real life scenarios and evaluating their performance and comprehensibility for humans. This work reveals practical obstacles when adopting XAI methods and gives qualitative estimates on how well different appr…
ENSTA Paris, Institut Polytechnique Paris（巴黎高等先进技术学院，巴黎综合理工学院）；Inria Flowers（法国国家信息与自动化研究所Flowers团队）；Renault（雷诺公司）；TECNALIA（TECNALIA研究中心）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟）；University of the Basque Country (UPV/EHU)（巴斯克大学）
Most of state of the art methods applied on time series consist of deep learning methods that are too complex to be interpreted. This lack of interpretability is a major drawback, as several applications in the real world are critical tasks, such as the medical field or the autonomous driving field. The explainability of models applied on time series has not gather much attention compared to the computer vision or the natural language processing fields. In this paper, we present an overview of existing explainable AI (XAI) methods applied on time series and illustrate the type of explanations …
A state-of-the-art systematic review on XAI applied to Prognostic and Health Management (PHM) of industrial asset is presented. This work provides an overview of the general trend of XAI in PHM, answers the question of accuracy versus explainability, the extent of human involvement, the explanation assessment and uncertainty quantification in PHM-XAI domain. Research articles associated with the subject, from 2015 to 2021 were selected from five known databases following PRISMA guidelines. Data was then extracted from the selected articles and examined. Several findings were synthesized. First…
This paper introduces the "grasp-ability test" as a "goodness" criteria by which to compare which explanation is more or less meaningful than others for users to understand the automated algorithmic data processing.
Explainable artificial intelligence (XAI) methods lack ground truth. In its place, method developers have relied on axioms to determine desirable properties for their explanations' behavior. For high stakes uses of machine learning that require explainability, it is not sufficient to rely on axioms as the implementation, or its usage, can fail to live up to the ideal. As a result, there exists active research on validating the performance of XAI methods. The need for validation is especially magnified in domains with a reliance on XAI. A procedure frequently used to assess their utility, and t…
Nanyang Technological University（南洋理工大学）；Alibaba Group Holding Limited（阿里巴巴集团控股有限公司）
Recently, artificial intelligence and machine learning in general have demonstrated remarkable performances in many tasks, from image processing to natural language processing, especially with the advent of deep learning. Along with research progress, they have encroached upon many different fields and disciplines. Some of them require high level of accountability and thus transparency, for example the medical sector. Explanations for machine decisions and predictions are thus needed to justify their reliability. This requires greater interpretability, which often means we need to understand t…
Explainable artificial intelligence (XAI) seeks to produce explanations for those machine learning methods which are deemed opaque. However, there is considerable disagreement about what this means and how to achieve it. Authors disagree on what should be explained (topic), to whom something should be explained (stakeholder), how something should be explained (instrument), and why something should be explained (goal). In this paper, I employ insights from means-end epistemology to structure the field. According to means-end epistemology, different means ought to be rationally adopted to achiev…
Meta Reality Labs（Meta现实实验室）；Purdue（普渡大学）；Aarhus Univ（奥胡斯大学）；Dartmouth（达特茅斯学院）；Cambridge（剑桥大学）
Explainable AI (XAI) has established itself as an important component of AI-driven interactive systems. With Augmented Reality (AR) becoming more integrated in daily lives, the role of XAI also becomes essential in AR because end-users will frequently interact with intelligent services. However, it is unclear how to design effective XAI experiences for AR. We propose XAIR, a design framework that addresses "when", "what", and "how" to provide explanations of AI output in AR. The framework was based on a multi-disciplinary literature review of XAI and HCI research, a large-scale survey probing …
Federal Office for Information Security (BSI)（联邦信息安全办公室）
The nondeterminism of Deep Learning (DL) training algorithms and its influence on the explainability of neural network (NN) models are investigated in this work with the help of image classification examples. To discuss the issue, two convolutional neural networks (CNN) have been trained and their results compared. The comparison serves the exploration of the feasibility of creating deterministic, robust DL models and deterministic explainable artificial intelligence (XAI) in practice. Successes and limitation of all here carried out efforts are described in detail. The source code of the atta…
Nowadays, deep neural networks are widely used in mission critical systems such as healthcare, self-driving vehicles, and military which have direct impact on human lives. However, the black-box nature of deep neural networks challenges its use in mission critical applications, raising ethical and judicial concerns inducing lack of trust. Explainable Artificial Intelligence (XAI) is a field of Artificial Intelligence (AI) that promotes a set of tools, techniques, and algorithms that can generate high-quality interpretable, intuitive, human-understandable explanations of AI decisions. In additi…
LUT University（拉彭兰塔-拉赫蒂工业大学）；Norwegian University of Science and Technology（挪威科技大学）；SINTEF Digital（挪威科技工业研究院数字研究所）
The final search query for the Systematic Literature Review (SLR) was conducted on 15th July 2022. Initially, we extracted 1707 journal and conference articles from the Scopus and Web of Science databases. Inclusion and exclusion criteria were then applied, and 58 articles were selected for the SLR. The findings show four dimensions that shape the AI explanation, which are format (explanation representation format), completeness (explanation should contain all required information, including the supplementary information), accuracy (information regarding the accuracy of the explanation), and c…
Breast Cancer Diagnosis: A Comprehensive Exploration of Explainable Artificial Intelligence (XAI) Techniques
Canadian Institute of Cybersecurity (CIC)（加拿大网络安全研究所）；University of New Brunswick（新不伦瑞克大学）；Dipartimento di Informatica, Università di Verona（威尼斯大学信息学院）；Università di Verona（威尼斯大学）；Department of Computer Science, School of Mathematics and Computer Science, Institute of Business Administration（计算机科学系，数学与计算机科学学院，商学院）；LIRIS, CNRS, Université Claude Bernard Lyon 1（LIRIS，国家科学研究中心，克莱尔伯恩大学 Lyon 1 分校）；LHC, UMR5516, Université de Saint-Etienne（LHC，UMR5516，圣艾蒂安大学）
This paper proposes an alternative approach to the basic taxonomy of explanations produced by explainable artificial intelligence techniques. Methods of Explainable Artificial Intelligence (XAI) were developed to answer the question why a certain prediction or estimation was made, preferably in terms easy to understand by the human agent. XAI taxonomies proposed in the literature mainly concentrate their attention on distinguishing explanations with respect to involving the human agent, which makes it complicated to provide a more mathematical approach to distinguish and compare different expl…
The XAI literature is decentralized, both in terminology and in publication venues, but recent years saw the community converge around keywords that make it possible to more reliably discover papers automatically. We use keyword search using the SemanticScholar API and manual curation to collect a well-formatted and reasonably comprehensive set of 5199 XAI papers, available at https://github.com/alonjacovi/XAI-Scholar . We use this collection to clarify and visualize trends about the size and scope of the literature, citation trends, cross-field trends, and collaboration trends. Overall, XAI i…
Centre for Forensic Science, University of Technology Sydney（悉尼科技大学法医科学中心）
Explainable Artificial Intelligence (XAI) is an emerging area of research in the field of Artificial Intelligence (AI). XAI can explain how AI obtained a particular solution (e.g., classification or object detection) and can also answer other "wh" questions. This explainability is not possible in traditional AI. Explainability is essential for critical applications, such as defense, health care, law and order, and autonomous driving vehicles, etc, where the know-how is required for trust and transparency. A number of XAI techniques so far have been purposed for such applications. This paper pr…
University of Hartford（哈特福德大学）；Tennessee Tech University（田纳西理工大学）；Edith Cowan University（伊迪丝·考恩大学）
The lack of explainability of a decision from an Artificial Intelligence (AI) based "black box" system/model, despite its superiority in many real-world applications, is a key stumbling block for adopting AI in many high stakes applications of different domain or industry. While many popular Explainable Artificial Intelligence (XAI) methods or approaches are available to facilitate a human-friendly explanation of the decision, each has its own merits and demerits, with a plethora of open challenges. We demonstrate popular XAI methods with a mutual case study/task (i.e., credit default predicti…
In the meantime, a wide variety of terminologies, motivations, approaches, and evaluation criteria have been developed within the research field of explainable artificial intelligence (XAI). With the amount of XAI methods vastly growing, a taxonomy of methods is needed by researchers as well as practitioners: To grasp the breadth of the topic, compare methods, and to select the right XAI method based on traits required by a specific use-case context. Many taxonomies for XAI methods of varying level of detail and depth can be found in the literature. While they often have a different focus, the…
Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science
Technische Universität Berlin（柏林工业大学）；ATB（莱布尼茨农业工程与生物经济研究所）；University of Leipzig（莱比锡大学）；University of Reading（雷丁大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；UiT the Arctic University of Norway（挪威北极大学）；University of Potsdam（波茨坦大学）
EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of…
A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototy…
Artificial intelligence has become pervasive across disciplines and fields, and biomedical image and signal processing is no exception. The growing and widespread interest on the topic has triggered a vast research activity that is reflected in an exponential research effort. Through study of massive and diverse biomedical data, machine and deep learning models have revolutionized various tasks such as modeling, segmentation, registration, classification and synthesis, outperforming traditional techniques. However, the difficulty in translating the results into biologically/clinically interpre…
This paper applies eXplainable Artificial Intelligence (XAI) methods to investigate the socioeconomic disparities in COVID patient mortality. An Extreme Gradient Boosting (XGBoost) prediction model is built based on a de-identified Austin area hospital dataset to predict the mortality of COVID-19 patients. We apply two XAI methods, Shapley Additive exPlanations (SHAP) and Locally Interpretable Model Agnostic Explanations (LIME), to compare the global and local interpretation of feature importance. This paper demonstrates the advantages of using XAI which shows the feature importance and decisi…
Explainable Artificial Intelligence (XAI) is a rising field in AI. It aims to produce a demonstrative factor of trust, which for human subjects is achieved through communicative means, which Machine Learning (ML) algorithms cannot solely produce, illustrating the necessity of an extra layer producing support to the model output. When approaching the medical field, we can see challenges arise when dealing with the involvement of human-subjects, the ideology behind trusting a machine to tend towards the livelihood of a human poses an ethical conundrum - leaving trust as the basis of the human-ex…
Saarland University（萨尔大学）；Technical University Dortmund（多特蒙德工业大学）；Institute of Intelligent Software（智能软件研究所）
Previous research in Explainable Artificial Intelligence (XAI) suggests that a main aim of explainability approaches is to satisfy specific interests, goals, expectations, needs, and demands regarding artificial systems (we call these stakeholders' desiderata) in a variety of contexts. However, the literature on XAI is vast, spreads out across multiple largely disconnected disciplines, and it often remains unclear how explainability approaches are supposed to achieve the goal of satisfying stakeholders' desiderata. This paper discusses the main classes of stakeholders calling for explainabilit…
Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
University of Gondar（贡德尔大学）；College of Engineering and Applied Science, University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校工程与应用科学学院）；Can Tho University（芹苴大学）；Ewha Womans University（梨花女子大学）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
Proceedings of The first international workshop on eXplainable AI for the Arts (XAIxArts)
University of the Arts London（伦敦艺术大学）；Queen Mary University of London（伦敦玛丽女王大学）；University of Nottingham（诺丁汉大学）；Central Conservatory of Music（中央音乐学院）；Tsinghua University（清华大学）；NYU Shanghai（上海纽约大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Many ML models are opaque to humans, producing decisions too complex for humans to easily understand. In response, explainable artificial intelligence (XAI) tools that analyze the inner workings of a model have been created. Despite these tools' strength in translating model behavior, critiques have raised concerns about the impact of XAI tools as a tool for `fairwashing` by misleading users into trusting biased or incorrect models. In this paper, we created a framework for evaluating explainable AI tools with respect to their capabilities for detecting and addressing issues of bias and fairne…
Technische Universität Berlin（柏林工业大学）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD——柏林学习与数据基础研究所）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）
In addition to the impressive predictive power of machine learning (ML) models, more recently, explanation methods have emerged that enable an interpretation of complex non-linear learning models such as deep neural networks. Gaining a better understanding is especially important e.g. for safety-critical ML applications or medical diagnostics etc. While such Explainable AI (XAI) techniques have reached significant popularity for classifiers, so far little attention has been devoted to XAI for regression models (XAIR). In this review, we clarify the fundamental conceptual differences of XAI for…
In recent years, the field of explainable AI (XAI) has produced a vast collection of algorithms, providing a useful toolbox for researchers and practitioners to build XAI applications. With the rich application opportunities, explainability is believed to have moved beyond a demand by data scientists or researchers to comprehend the models they develop, to an essential requirement for people to trust and adopt AI deployed in numerous domains. However, explainability is an inherently human-centric property and the field is starting to embrace human-centered approaches. Human-computer interactio…
Georgia Institute of Technology（佐治亚理工学院）；Microsoft Research（微软研究院）
Explainable AI (XAI) systems are sociotechnical in nature; thus, they are subject to the sociotechnical gap--divide between the technical affordances and the social needs. However, charting this gap is challenging. In the context of XAI, we argue that charting the gap improves our problem understanding, which can reflexively provide actionable insights to improve explainability. Utilizing two case studies in distinct domains, we empirically derive a framework that facilitates systematic charting of the sociotechnical gap by connecting AI guidelines in the context of XAI and elucidating how to …
Mepco Schlenk Engineering College（梅普科施伦克工程学院）；Pusan National University（釜山国立大学）；Shenzhen University（深圳大学）；Zhejiang University（浙江大学）；Zhejiang Lab（之江实验室）；Southeast University（东南大学）
Explainable Artificial Intelligence (XAI) is transforming the field of Artificial Intelligence (AI) by enhancing the trust of end-users in machines. As the number of connected devices keeps on growing, the Internet of Things (IoT) market needs to be trustworthy for the end-users. However, existing literature still lacks a systematic and comprehensive survey work on the use of XAI for IoT. To bridge this lacking, in this paper, we address the XAI frameworks with a focus on their characteristics and support for IoT. We illustrate the widely-used XAI services for IoT applications, such as securit…
Proceedings of The third international workshop on eXplainable AI for the Arts (XAIxArts)
University of the Arts London（伦敦艺术大学）；Queen Mary University of London（伦敦玛丽女王大学）；Concordia University（Concordia大学）；Towson University（Towson大学）；Tsinghua University（清华大学）；University of Utah（乌得勒支大学）；Kingston University（Kingston大学）；University of Edinburgh（爱丁堡大学）；University of Nottingham（特兰特大学）
Recent years have seen a surge of interest in the field of explainable AI (XAI), with a plethora of algorithms proposed in the literature. However, a lack of consensus on how to evaluate XAI hinders the advancement of the field. We highlight that XAI is not a monolithic set of technologies -- researchers and practitioners have begun to leverage XAI algorithms to build XAI systems that serve different usage contexts, such as model debugging and decision-support. Algorithmic research of XAI, however, often does not account for these diverse downstream usage contexts, resulting in limited effecti…
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1)…
Explainable Artificial Intelligence (XAI) techniques are frequently required by users in many AI systems with the goal of understanding complex models, their associated predictions, and gaining trust. While suitable for some specific tasks during development, their adoption by organisations to enhance trust in machine learning systems has unintended consequences. In this paper we discuss XAI's limitations in deployment and conclude that transparency alongside with rigorous validation are better suited to gaining trust in AI systems.
Evolutionary Computation and Explainable AI: A Roadmap to Understandable Intelligent Systems
Queen’s University（女王大学）；Newcastle University（纽卡斯尔大学）；University of Stirling（斯特灵大学）；University of Parma（帕尔马大学）；Robert Gordon University（罗伯特戈登大学）；University of Trento（特伦托大学）；Universiteit Leiden（莱顿大学）；University of Exeter（埃克塞特大学）
IBM Research AI（IBM研究院AI）；IBM Design（IBM设计）；IBM Research（IBM研究院）
A pervasive design issue of AI systems is their explainability--how to provide appropriate information to help users understand the AI. The technical field of explainable AI (XAI) has produced a rich toolbox of techniques. Designers are now tasked with the challenges of how to select the most suitable XAI techniques and translate them into UX solutions. Informed by our previous work studying design challenges around XAI UX, this work proposes a design process to tackle these challenges. We review our and related prior work to identify requirements that the process should fulfill, and according…
Proceedings of The second international workshop on eXplainable AI for the Arts (XAIxArts)
University of the Arts London（伦敦艺术大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；University of Nottingham（诺丁汉大学）；Kingston University（金斯顿大学）；University of Edinburgh（爱丁堡大学）；Central Conservatory of Music（中央音乐学院）；Tsinghua University（清华大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Towson University（陶森大学）；University of Utah（犹他大学）；Concordia University（康考迪亚大学）
