Transparency of Machine Learning models used for decision support in various industries becomes essential for ensuring their ethical use. To that end, feature attribution methods such as SHAP (SHapley Additive exPlanations) are widely used to explain the predictions of black-box machine learning models to customers and developers. However, a parallel trend has been to train machine learning models in collaboration with other data holders without accessing their data. Such models, trained over horizontally or vertically partitioned data, present a challenge for explainable AI because the explaining party may have a biased view of background data or a partial view of the feature space. As a result, explanations obtained from different participants of distributed machine learning might not be consistent with one another, undermining trust in the product. This paper presents an Explainable Data Collaboration Framework based on a model-agnostic additive feature attribution algorithm (KernelSHAP) and Data Collaboration method of privacy-preserving distributed machine learning. In particular, we present three algorithms for different scenarios of explainability in Data Collaboration and verify their consistency with experiments on open-access datasets. Our results demonstrated a significant (by at least a factor of 1.75) decrease in feature attribution discrepancies among the users of distributed machine learning.
BACON: A fully explainable AI model with graded logic for decision making problems
Department of Information Systems University of Maryland, Baltimore County (UMBC)（信息系统系大学马里兰大学巴尔的摩县（UMBC））；Department of Computer Science San Francisco State University（计算机科学系旧金山州立大学）；Department of Information Systems UMBC（信息系统系大学马里兰大学巴尔的摩县（UMBC））
Although modern machine learning and deep learning methods allow for complex and in-depth data analytics, the predictive models generated by these methods are often highly complex, and lack transparency. Explainable AI (XAI) methods are used to improve the interpretability of these complex models, and in doing so improve transparency. However, the inherent fitness of these explainable methods can be hard to evaluate. In particular, methods to evaluate the fidelity of the explanation to the underlying black box require further development, especially for tabular data. In this paper, we (a) prop…
Financial Fraud Detection Using Explainable AI and Stacking Ensemble Methods
Computer Science Department, College of Computers and Information Technology, Taif University（计算机科学系，计算机与信息科技学院，泰夫大学）；Department of Computer Science, Taif University（计算机科学系，泰夫大学）
Explainable Artificial Intelligence (XAI) techniques are frequently required by users in many AI systems with the goal of understanding complex models, their associated predictions, and gaining trust. While suitable for some specific tasks during development, their adoption by organisations to enhance trust in machine learning systems has unintended consequences. In this paper we discuss XAI's limitations in deployment and conclude that transparency alongside with rigorous validation are better suited to gaining trust in AI systems.
Centre for Forensic Science, University of Technology Sydney（悉尼科技大学法医科学中心）
Explainable Artificial Intelligence (XAI) is an emerging area of research in the field of Artificial Intelligence (AI). XAI can explain how AI obtained a particular solution (e.g., classification or object detection) and can also answer other "wh" questions. This explainability is not possible in traditional AI. Explainability is essential for critical applications, such as defense, health care, law and order, and autonomous driving vehicles, etc, where the know-how is required for trust and transparency. A number of XAI techniques so far have been purposed for such applications. This paper pr…
Predictive process analytics focuses on predicting the future states of running instances of a business process. While advanced machine learning techniques have been used to increase accuracy of predictions, the resulting predictive models lack transparency. Current explainable machine learning methods, such as LIME and SHAP, can be used to interpret black box models. However, it is unclear how fit for purpose these methods are in explaining process predictive models. In this paper, we draw on evaluation measures used in the field of explainable AI and propose functionally-grounded evaluation …
Lack of transparency has been the Achilles heal of Neural Networks and their wider adoption in industry. Despite significant interest this shortcoming has not been adequately addressed. This study proposes a novel framework called Hide-and-Seek (HnS) for training Interpretable Neural Networks and establishes a theoretical foundation for exploring and comparing similar ideas. Extensive experimentation indicates that a high degree of interpretability can be imputed into Neural Networks, without sacrificing their predictive power.
Fraunhofer FIT（弗劳恩霍夫应用信息技术研究所）；University of Cologne（科隆大学）；RWTH Aachen University（亚琛工业大学）；Elsevier（爱思唯尔）；ZBMED - Information Center for Life Sciences（ZBMED生命科学信息中心）
Artificial intelligence (AI) systems utilizing deep neural networks (DNNs) and machine learning (ML) algorithms are widely used for solving important problems in bioinformatics, biomedical informatics, and precision medicine. However, complex DNNs or ML models, which are often perceived as opaque and black-box, can make it difficult to understand the reasoning behind their decisions. This lack of transparency can be a challenge for both end-users and decision-makers, as well as AI developers. Additionally, in sensitive areas like healthcare, explainability and accountability are not only desir…
Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirements
Institute of Software Engineering University of Stuttgart Stuttgart, Germany Email；Institute of Software Engineering University of Stuttgart Stuttgart, Germany
A Unified Framework for Evaluating and Enhancing the Transparency of Explainable AI Methods via Perturbation-Gradient Consensus Attribution
Department of Computer Science, American International University-Bangladesh（美国国际大学-孟加拉国计算机科学系）；Thomas Lord Department of Computer Science, Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院托马斯·洛德计算机科学系）；Department of Computer Science and Engineering, Techno International New Town（Techno International New Town计算机科学与工程系）
Center for Artificial Intelligence Research（人工智能研究中心）；University of Agder（阿格德大学）
The past decade has seen significant progress in artificial intelligence (AI), which has resulted in algorithms being adopted for resolving a variety of problems. However, this success has been met by increasing model complexity and employing black-box AI models that lack transparency. In response to this need, Explainable AI (XAI) has been proposed to make AI more transparent and thus advance the adoption of AI in critical domains. Although there are several reviews of XAI topics in the literature that identified challenges and potential research directions in XAI, these challenges and resear…
Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
University of Gondar（贡德尔大学）；College of Engineering and Applied Science, University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校工程与应用科学学院）；Can Tho University（芹苴大学）；Ewha Womans University（梨花女子大学）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
University of Florida（佛罗里达大学）；University of Texas in Dallas（德克萨斯大学达拉斯分校）
Explainable machine learning and artificial intelligence models have been used to justify a model's decision-making process. This added transparency aims to help improve user performance and understanding of the underlying model. However, in practice, explainable systems face many open questions and challenges. Specifically, designers might reduce the complexity of deep learning models in order to provide interpretability. The explanations generated by these simplified models, however, might not accurately justify and be truthful to the model. This can further add confusion to the users as the…
HU University of Applied Sciences Utrecht（乌得勒支应用科学大学）；De Nederlandsche Bank（荷兰中央银行）
Explainable artificial intelligence (xAI) is seen as a solution to making AI systems less of a black box. It is essential to ensure transparency, fairness, and accountability, which are especially paramount in the financial sector. The aim of this study was a preliminary investigation of the perspectives of supervisory authorities and regulated entities regarding the application of xAI in the fi-nancial sector. Three use cases (consumer credit, credit risk, and anti-money laundering) were examined using semi-structured interviews at three banks and two supervisory authorities in the Netherland…
A Comprehensive Review on Financial Explainable AI
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；Asian Institute of Digital Finance at NUS（新加坡国立大学亚洲数字金融研究所）；Imperial College London（帝国理工学院）
Generalizable and Explainable Deep Learning for Medical Image Computing: An Overview
School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；École de Technologie Supérieure (ETS)（高等技术学院）；College of Computer Science and Engineering, Taibah University（塔伊布大学计算机科学与工程学院）；College of Computer Science and Engineering, Jeddah University（吉达大学计算机科学与工程学院）
This article aims to provide a theoretical account and corresponding paradigm for analysing how explainable artificial intelligence (XAI) influences people's behaviour and cognition. It uses insights from research on behaviour change. Two notable frameworks for thinking about behaviour change techniques are nudges - aimed at influencing behaviour - and boosts - aimed at fostering capability. It proposes that local and concept-based explanations are more adjacent to nudges, while global and counterfactual explanations are more adjacent to boosts. It outlines a method for measuring XAI influence…
Enhancing transparency in AI-powered customer engagement
Pegasystems（Pegasystems公司）；Oracle（甲骨文公司）；EMC（易安信公司）；University of California at Berkeley（加州大学伯克利分校）；University of Massachusetts at Amherst（马萨诸塞大学阿默斯特分校）
Computer Science Department, Arizona State University（亚利桑那州立大学计算机科学系）；IBM T. J. Watson Research Center（IBM T.J.沃森研究中心）
In this paper, we report on the visualization capabilities of an Explainable AI Planning (XAIP) agent that can support human in the loop decision making. Imposing transparency and explainability requirements on such agents is especially important in order to establish trust and common ground with the end-to-end automated planning system. Visualizing the agent's internal decision-making processes is a crucial step towards achieving this. This may include externalizing the "brain" of the agent -- starting from its sensory inputs, to progressively higher order decisions made by it in order to dri…
Cybersecurity is a domain where the data distribution is constantly changing with attackers exploring newer patterns to attack cyber infrastructure. Intrusion detection system is one of the important layers in cyber safety in today's world. Machine learning based network intrusion detection systems started showing effective results in recent years. With deep learning models, detection rates of network intrusion detection system are improved. More accurate the model, more the complexity and hence less the interpretability. Deep neural networks are complex and hard to interpret which makes diffi…
Georgia Institute of Technology（佐治亚理工学院）；IBM Research AI（IBM研究院人工智能部门）
As AI-powered systems increasingly mediate consequential decision-making, their explainability is critical for end-users to take informed and accountable actions. Explanations in human-human interactions are socially-situated. AI systems are often socio-organizationally embedded. However, Explainable AI (XAI) approaches have been predominantly algorithm-centered. We take a developmental step towards socially-situated XAI by introducing and exploring Social Transparency (ST), a sociotechnically informed perspective that incorporates the socio-organizational context into explaining AI-mediated d…
College of Computing and Informatics, Drexel University（德雷塞尔大学计算与信息学院）
We present a focused analysis of user studies in explainable artificial intelligence (XAI) entailing qualitative investigation. We draw on social science corpora to suggest ways for improving the rigor of studies where XAI researchers use observations, interviews, focus groups, and/or questionnaires to capture qualitative data. We contextualize the presentation of the XAI papers included in our analysis according to the components of rigor described in the qualitative research literature: 1) underlying theories or frameworks, 2) methodological approaches, 3) data collection methods, and 4) dat…
Artificial intelligence (AI) is gaining momentum, and its importance for the future of work in many areas, such as medicine and banking, is continuously rising. However, insights on the effective collaboration of humans and AI are still rare. Typically, AI supports humans in decision-making by addressing human limitations. However, it may also evoke human bias, especially in the form of automation bias as an over-reliance on AI advice. We aim to shed light on the potential to influence automation bias by explainable AI (XAI). In this pre-test, we derive a research model and describe our study …
Collaboration is identified as a required and necessary skill for students to be successful in the fields of Science, Technology, Engineering and Mathematics (STEM). However, due to growing student population and limited teaching staff it is difficult for teachers to provide constructive feedback and instill collaborative skills using instructional methods. Development of simple and easily explainable machine-learning-based automated systems can help address this problem. Improving upon our previous work, in this paper we propose using simple temporal-CNN deep-learning models to assess student…
