This paper presents flood prediction models for the state of Kerala in India by analyzing the monthly rainfall data and applying machine learning algorithms including Logistic Regression, K-Nearest Neighbors, Decision Trees, Random Forests, and Support Vector Machine. Although these models have shown high accuracy prediction of the occurrence of flood in a particular year, they do not quantitatively and qualitatively explain the prediction decision. This paper shows how the background features are learned that contributed to the prediction decision and further extended to explain the inner workings with the development of explainable artificial intelligence modules. The obtained results have confirmed the validity of the findings uncovered by the explainer modules basing on the historical flood monthly rainfall data in Kerala.
A Transformer variant for multi-step forecasting of water level and hydrometeorological sensitivity analysis based on explainable artificial intelligence technology
School of Internet, Anhui University（安徽大学互联网学院）；National Engineering Research Center for Agro-Ecological Big Data Analysis & Application, Anhui University（安徽大学农业生态大数据分析与应用国家工程研究中心）；Forschungszentrum Jülich GmbH, Institute of Energy and Climate Research (IEK)（于利希研究中心能源与气候研究所）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）
We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP.…
This paper contributes to automating medical image segmentation by proposing generative adversarial network-based models to segment both polyps and instruments in endoscopy images. A major contribution of this work is to provide explanations for the predictions using a layer-wise relevance propagation approach designating which input image pixels are relevant to the predictions and to what extent. On the polyp segmentation task, the models achieved 0.84 of accuracy and 0.46 on Jaccard index. On the instrument segmentation task, the models achieved 0.96 of accuracy and 0.70 on Jaccard index. Th…
Cardiff University（卡迪夫大学）；Crime and Security Research Institute（犯罪与安全研究院）；School of Computer Science & Informatics（计算机科学与信息学院）；School of Engineering（工程学院）
Current techniques for explainable AI have been applied with some success to image processing. The recent rise of research in video processing has called for similar work n deconstructing and explaining spatio-temporal models. While many techniques are designed for 2D convolutional models, others are inherently applicable to any input domain. One such body of work, deep Taylor decomposition, propagates relevance from the model output distributively onto its input and thus is not restricted to image processing models. However, by exploiting a simple technique that removes motion information, we…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）
In this paper, we tackle the problem of explanations in a deep-learning based model for recommendations by leveraging the technique of layer-wise relevance propagation. We use a Deep Convolutional Neural Network to extract relevant features from the input images before identifying similarity between the images in feature space. Relationships between the images are identified by the model and layer-wise relevance propagation is used to infer pixel-level details of the images that may have significantly informed the model's choice. We evaluate our method on an Amazon products dataset and demonst…
Context-Aware Concept Distillation for Trustworthy Flood Prediction
School of Computer Science and Engineering, The Hebrew University of Jerusalem（耶路撒冷希伯来大学计算机科学与工程学院）；Institute of Earth Sciences, The Hebrew University of Jerusalem（耶路撒冷希伯来大学地球科学研究所）；Hebrew University Business School, The Hebrew University of Jerusalem（耶路撒冷希伯来大学商学院）
Epilepsy which is characterized by seizures is studied using EEG signals by recording the electrical activity of the brain. Different types of communication between different parts of the brain are characterized by many state of the art connectivity measures which can be directed and undirected. We propose to employ a set of undirected (spectral matrix, the inverse of the spectral matrix, coherence, partial coherence, and phaselocking value) and directed features (directed coherence, the partial directed coherence) to learn a deep neural network that detects whether a particular data window be…
University of Bamberg（班贝格大学）；IBM Deutschland GmbH（IBM德国有限公司）；fortiss GmbH（福蒂斯有限公司）
Artificial Intelligence and Digital Twins play an integral role in driving innovation in the domain of intelligent driving. Long short-term memory (LSTM) is a leading driver in the field of lane change prediction for manoeuvre anticipation. However, the decision-making process of such models is complex and non-transparent, hence reducing the trustworthiness of the smart solution. This work presents an innovative approach and a technical implementation for explaining lane change predictions of layer normalized LSTMs using Layer-wise Relevance Propagation (LRP). The core implementation includes …
Convolutional neural networks (CNN) are known for their excellent feature extraction capabilities to enable the learning of models from data, yet are used as black boxes. An interpretation of the convolutional filtres and associated features can help to establish an understanding of CNN to distinguish various classes. In this work, we focus on the explainability of a CNN model called as cnnexplain that is used for Covid-19 and non-Covid-19 classification with a focus on the interpretability of features by the convolutional filters, and how these features contribute to classification. Specifica…
University at Buffalo, State University of New York（纽约州立大学布法罗分校）；University of Oregon（俄勒冈大学）
A framework is presented to extract and understand decision-making information from a deep neural network (DNN) classifier of jet substructure tagging techniques. The general method studied is to provide expert variables that augment inputs ("eXpert AUGmented" variables, or XAUG variables), then apply layerwise relevance propagation (LRP) to networks both with and without XAUG variables. The XAUG variables are concatenated with the intermediate layers after network-specific operations (such as convolution or recurrence), and used in the final layers of the network. The results of comparing net…
UiT the Arctic University of Norway（挪威北极大学（特罗姆瑟大学））；University Hospital of North Norway (UNN)（北挪威大学医院（UNN））
Deep learning-based support systems have demonstrated encouraging results in numerous clinical applications involving the processing of time series data. While such systems often are very accurate, they have no inherent mechanism for explaining what influenced the predictions, which is critical for clinical tasks. However, existing explainability techniques lack an important component for trustworthy and reliable decision support, namely a notion of uncertainty. In this paper, we address this lack of uncertainty by proposing a deep ensemble approach where a collection of DNNs are trained indep…
A Unified Framework for Uncertainty-Aware Explainable Artificial Intelligence: A Case Study in Power Quality Disturbance Classification
School of Engineering, Deakin University（德肯大学工程学院）；Faculty of Mathematics and Computer Science, FernUniversität in Hagen（哈根应用科学大学数学与计算机科学学院）；Department of Computing Technologies, Swinburne University of Technology（斯威本科技大学计算技术学院）
Tilburg University（蒂尔堡大学）；Universidad de Talca（塔尔卡大学）；Eindhoven University of Technology（埃因霍温理工大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；University of Miami（迈阿密大学）
Machine learning solutions for pattern classification problems are nowadays widely deployed in society and industry. However, the lack of transparency and accountability of most accurate models often hinders their safe use. Thus, there is a clear need for developing explainable artificial intelligence mechanisms. There exist model-agnostic methods that summarize feature contributions, but their interpretability is limited to predictions made by black-box models. An open challenge is to develop models that have intrinsic interpretability and produce their own explanations, even for classes of m…
AudioMNIST: Exploring Explainable Artificial Intelligence for Audio Analysis on a Simple Benchmark
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD——柏林学习与数据基础研究所）
Cybersecurity is a domain where there is constant change in patterns of attack, and we need ways to make our Cybersecurity systems more adaptive to handle new attacks and categorize for appropriate action. We present a novel approach to handle the alarm flooding problem faced by Cybersecurity systems like security information and event management (SIEM) and intrusion detection (IDS). We apply a zero-shot learning method to machine learning (ML) by leveraging explanations for predictions of anomalies generated by a ML model. This approach has huge potential to auto detect alarm labels generated…
Brunel University London（伦敦布鲁内尔大学）；Technical University of Darmstadt（达姆施塔特工业大学）；American International University-Bangladesh（孟加拉国国际大学）；Dhaka University of Engineering and Technology（达卡工程技术大学）；George Mason University（乔治梅森大学）
Clausthal University of Technology（克劳斯塔尔工业大学）；German Center for Neurodegenerative Diseases（德国神经退行性疾病研究中心）；University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）；Genevention GmbH（Genevention有限责任公司）
The lack of well-structured metadata annotations complicates there-usability and interpretation of the growing amount of publicly available RNA expression data. The machine learning-based prediction of metadata(data augmentation) can considerably improve the quality of expression data annotation. In this study,we systematically benchmark deep learning (DL) and random forest (RF)-based metadata augmentation of tissue, age, and sex using small RNA (sRNA) expression profiles. We use 4243 annotated sRNA-Seq samples from the small RNA expression atlas (SEA) database to train and test the augmentati…
University of Cagliari（卡利亚里大学）；Pluribus One S.r.l.（Pluribus One有限责任公司）
One of the most common causes of lack of continuity of online systems stems from a widely popular Cyber Attack known as Distributed Denial of Service (DDoS), in which a network of infected devices (botnet) gets exploited to flood the computational capacity of services through the commands of an attacker. This attack is made by leveraging the Domain Name System (DNS) technology through Domain Generation Algorithms (DGAs), a stealthy connection strategy that yet leaves suspicious data patterns. To detect such threats, advances in their analysis have been made. For the majority, they found Machin…
Universidad Autonoma de Madrid（马德里自治大学）；Laboratoire des Sciences du Numérique de Nantes（南特数字科学实验室）；National Institute of Informatics（国立信息学研究所）
Machine learning methods are growing in relevance for biometrics and personal information processing in domains such as forensics, e-health, recruitment, and e-learning. In these domains, white-box (human-readable) explanations of systems built on machine learning methods can become crucial. Inductive Logic Programming (ILP) is a subfield of symbolic AI aimed to automatically learn declarative theories about the process of data. Learning from Interpretation Transition (LFIT) is an ILP technique that can learn a propositional logic theory equivalent to a given black-box system (under certain co…
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering（计算机系统与生物信息学研究单位，工程系）；Research Unit of Radiation Oncology, Department of Medicine and Surgery（放射肿瘤学研究单位，医学与外科系）；Research Unit of Anatomical Pathology, Department of Medicine and Surgery（病理学研究单位，医学与外科系）
Purpose: To explore the potential of Explainable Machine Learning in the prediction and detection of drivers of cleared homicides at the national- and state-levels in the United States. Methods: First, nine algorithmic approaches are compared to assess the best performance in predicting cleared homicides country-wise, using data from the Murder Accountability Project. The most accurate algorithm among all (XGBoost) is then used for predicting clearance outcomes state-wise. Second, SHAP, a framework for Explainable Artificial Intelligence, is employed to capture the most important features in e…
Institute for Informatics and Telematics (IIT), National Research Council of Italy (CNR)（意大利国家研究委员会（CNR）信息学与电信研究所（IIT））
Nowadays Artificial Intelligence (AI) has become a fundamental component of healthcare applications, both clinical and remote, but the best performing AI systems are often too complex to be self-explaining. Explainable AI (XAI) techniques are defined to unveil the reasoning behind the system's predictions and decisions, and they become even more critical when dealing with sensitive and personal health data. It is worth noting that XAI has not gathered the same attention across different research areas and data types, especially in healthcare. In particular, many clinical and remote health appl…
The University of Arizona（亚利桑那大学）；Virginia Tech（弗吉尼亚理工大学）；University of Cincinnati（辛辛那提大学）
A longstanding challenge surrounding deep learning algorithms is unpacking and understanding how they make their decisions. Explainable Artificial Intelligence (XAI) offers methods to provide explanations of internal functions of algorithms and reasons behind their decisions in ways that are interpretable and understandable to human users. . Numerous XAI approaches have been developed thus far, and a comparative analysis of these strategies seems necessary to discern their relevance to clinical prediction models. To this end, we first implemented two prediction models for short- and long-term …
Università degli Studi di Napoli Federico II（那不勒斯费德里科二世大学）
This work proposes a novel general framework, in the context of eXplainable Artificial Intelligence (XAI), to construct explanations for the behaviour of Machine Learning (ML) models in terms of middle-level features. One can isolate two different ways to provide explanations in the context of XAI: low and middle-level explanations. Middle-level explanations have been introduced for alleviating some deficiencies of low-level explanations such as, in the context of image classification, the fact that human users are left with a significant interpretive burden: starting from low-level explanatio…
EXplainable AI (XAI) methods have been proposed to interpret how a deep neural network predicts inputs through model saliency explanations that highlight the parts of the inputs deemed important to arrive a decision at a specific target. However, it remains challenging to quantify correctness of their interpretability as current evaluation approaches either require subjective input from humans or incur high computation cost with automated evaluation. In this paper, we propose backdoor trigger patterns--hidden malicious functionalities that cause misclassification--to automate the evaluation of…
A Machine Learning Pipeline for Multiple Sclerosis Biomarker Discovery: Comparing explainable AI and Traditional Statistical Approaches
Department of Computer Science , University of Pisa（计算机科学系，比萨大学）；Department of Clinical and Biological Sciences, University of Torino（临床与生物科学系，都灵大学）；Department of Computer Sciences, University of Torino（计算机科学系，都灵大学）；Department of Computer Science and Engineering, University of Bologna（计算机科学与工程系，博洛尼亚大学）
该研究构建整合八个 PBMC 微阵列数据集的机器学习流水线，用 XGBoost 与 SHAP 发现 MS 生物标志物，并与 DEA 比较，证明二者互补且可揭示相关疾病通路。
Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI
Tübingen AI Center, University of Tübingen Germany（图宾根人工智能中心、图宾根大学德国）；Fraunhofer Heinrich Hertz Institute Germany（弗劳恩霍夫海因里希·赫兹研究所德国）；Information and Interaction Design, Yonsei University South Korea（信息与交互设计，延世大学韩国）
Imperial College London（伦敦帝国理工学院）；Princeton University（普林斯顿大学）；NOAA/Geophysical Fluid Dynamics Laboratory（美国国家海洋和大气管理局/地球物理流体动力学实验室）；University of Washington（华盛顿大学）；Institut Pierre-Simon Laplace（皮埃尔-西蒙·拉普拉斯研究所）；IRD（法国发展研究院）；Sorbonne Université（索邦大学）；LOCEAN-IPSL（皮埃尔-西蒙·拉普拉斯环境研究所海洋学与气候实验室）；CNRS（法国国家科学研究中心）；Laboratoire des Sciences du Climat et de l’Environnement（气候与环境科学实验室）；CEA Saclay（法国原子能和替代能源委员会萨克雷中心）
The trustworthiness of neural networks is often challenged because they lack the ability to express uncertainty and explain their skill. This can be problematic given the increasing use of neural networks in high stakes decision-making such as in climate change applications. We address both issues by successfully implementing a Bayesian Neural Network (BNN), where parameters are distributions rather than deterministic, and applying novel implementations of explainable AI (XAI) techniques. The uncertainty analysis from the BNN provides a comprehensive overview of the prediction more suited to p…
xCG: Explainable Cell Graphs for Survival Prediction in Non-Small Cell Lung Cancer
Aignostics, Germany（艾格诺斯蒂克斯公司（德国））；Technische Universität Berlin, Germany（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data, Germany（柏林学习与数据基础研究所）；Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA), Germany（康拉德·楚泽学习与智能系统卓越学院）；Charité – Universitätsmedizin Berlin, Germany（柏林夏里特医学院）；Ludwig-Maximilians-Universität München, Germany（慕尼黑大学）；Bavarian Cancer Research Center (BZKF), Germany（巴伐利亚癌症研究中心）；Max-Planck Institute for Informatics, Germany（马克斯·普朗克信息学研究所）；Korea University, Republic of Korea（高丽大学）
In modern business processes, the amount of data collected has increased substantially in recent years. Because this data can potentially yield valuable insights, automated knowledge extraction based on process mining has been proposed, among other techniques, to provide users with intuitive access to the information contained therein. At present, the majority of technologies aim to reconstruct explicit business process models. These are directly interpretable but limited concerning the integration of diverse and real-valued information sources. On the other hand, Machine Learning (ML) benefit…
There has been a recent surge in the field of Explainable AI (XAI) which tackles the problem of providing insights into the behavior of black-box machine learning models. Within this field, \textit{feature attribution} encompasses methods which assign relevance scores to input features and visualize them as a heatmap. Designing flexible accelerators for multiple such algorithms is challenging since the hardware mapping of these algorithms has not been studied yet. In this work, we first analyze the dataflow of gradient backpropagation based feature attribution algorithms to determine the resou…
Fraunhofer HHI（弗劳恩霍夫海因里希·赫兹研究所）；Humboldt University Berlin（柏林洪堡大学）
The task of detecting morphed face images has become highly relevant in recent years to ensure the security of automatic verification systems based on facial images, e.g. automated border control gates. Detection methods based on Deep Neural Networks (DNN) have been shown to be very suitable to this end. However, they do not provide transparency in the decision making and it is not clear how they distinguish between genuine and morphed face images. This is particularly relevant for systems intended to assist a human operator, who should be able to understand the reasoning. In this paper, we ta…
University of Trento（特伦托大学）；Temple University（天普大学）
Current models on Explainable Artificial Intelligence (XAI) have shown an evident and quantified lack of reliability for measuring feature-relevance when statistically entangled features are proposed for training deep classifiers. There has been an increase in the application of Deep Learning in clinical trials to predict early diagnosis of neuro-developmental disorders, such as Autism Spectrum Disorder (ASD). However, the inclusion of more reliable saliency-maps to obtain more trustworthy and interpretable metrics using neural activity features is still insufficiently mature for practical app…
PSO-XAI: A PSO-Enhanced Explainable AI Framework for Reliable Breast Cancer Detection
Department of Information and Communication Engineering, Noakhali Science and Technology University（信息与通信工程系，诺阿克利科学与技术大学）；Department of Computer Science & Engineering, Netrokona University（计算机科学与工程系，纳特罗克纳大学）；Department of Computer and Communication Engineering, International Islamic University Chittagong（计算机与通信工程系，国际伊斯兰大学查塔格ONG）；Department of Mechatronics and Industrial Engineering, Chittagong University of Engineering and Technology（机电与工业工程系，查塔格ONG工程与技术大学）；School of Computing Sciences and Computer Engineering, University of Southern Mississippi（计算科学与计算机工程学院，密西西比州立大学）；AI & Digital Health Technology, Artificial Intelligence and Cyber Futures Institute, Charles Sturt University（人工智能与数字健康技术，人工智能与网络未来研究院，查尔斯·斯图尔特大学）；AI & Digital Health Technology, Rural Health Research Institute, Charles Sturt University（人工智能与数字健康技术，农村健康研究学院，查尔斯·斯图尔特大学）
From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
Fraunhofer Heinrich-Hertz-Institute（弗劳恩霍夫海因里希-赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（BIFOLD – 柏林学习与数据基础研究所）
