State-of-the-art deep neural networks (DNNs) are highly effective in solving many complex real-world problems. However, these models are vulnerable to adversarial perturbation attacks, and despite the plethora of research in this domain, to this day, adversaries still have the upper hand in the cat and mouse game of adversarial example generation methods vs. detection and prevention methods. In this research, we present a novel detection method that uses Shapley Additive Explanations (SHAP) values computed for the internal layers of a DNN classifier to discriminate between normal and adversarial inputs. We evaluate our method by building an extensive dataset of adversarial examples over the popular CIFAR-10 and MNIST datasets, and training a neural network-based detector to distinguish between normal and adversarial inputs. We evaluate our detector against adversarial examples generated by diverse state-of-the-art attacks and demonstrate its high detection accuracy and strong generalization ability to adversarial inputs generated with different attack methods.
Pandit Deendayal Petroleum University（潘迪特·迪恩达亚尔石油大学）；Confirm SFI Centre for Smart Manufacturing（Confirm SFI智能制造中心）；Data Science Institute（数据科学研究所）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）
With the increasing availability of structured and unstructured data and the swift progress of analytical techniques, Artificial Intelligence (AI) is bringing a revolution to the healthcare industry. With the increasingly indispensable role of AI in healthcare, there are growing concerns over the lack of transparency and explainability in addition to potential bias encountered by predictions of the model. This is where Explainable Artificial Intelligence (XAI) comes into the picture. XAI increases the trust placed in an AI system by medical practitioners as well as AI researchers, and thus, ev…
As machine learning models grow more complex and their applications become more high-stakes, tools for explaining model predictions have become increasingly important. This has spurred a flurry of research in model explainability and has given rise to feature attribution methods such as LIME and SHAP. Despite their widespread use, evaluating and comparing different feature attribution methods remains challenging: evaluations ideally require human studies, and empirical evaluation metrics are often data-intensive or computationally prohibitive on real-world datasets. In this work, we address th…
Federal University of Pará（帕拉联邦大学）；Federal Institute of Pará（帕拉联邦学院）；Montpellier University（蒙彼利埃大学）；Vale Institute of Technology（淡水河谷研究所）
Strategies based on Explainable Artificial Intelligence - XAI have emerged in computing to promote a better understanding of predictions made by black box models. Most XAI measures used today explain these types of models, generating attribute rankings aimed at explaining the model, that is, the analysis of Attribute Importance of Model. There is no consensus on which XAI measure generates an overall explainability rank. For this reason, several proposals for tools have emerged (Ciu, Dalex, Eli5, Lofo, Shap and Skater). An experimental benchmark of explainable AI techniques capable of producin…
IBM Research Europe（IBM欧洲研究院）；IBM GBS Germany（IBM德国全球企业咨询服务部）；IBM GBS Switzerland（IBM瑞士全球企业咨询服务部）；Fondazione IRCCS Ca’ Granda, Ospedale Maggiore Policlinico（Ca' Granda基金会IRCCS马焦雷综合医院）；Humanitas University（人道大学）；IRCCS - Humanitas Research Hospital（IRCCS人道研究医院）；IBM GBS Italy（IBM意大利全球企业咨询服务部）
We develop various AI models to predict hospitalization on a large (over 110$k$) cohort of COVID-19 positive-tested US patients, sourced from March 2020 to February 2021. Models range from Random Forest to Neural Network (NN) and Time Convolutional NN, where combination of the data modalities (tabular and time dependent) are performed at different stages (early vs. model fusion). Despite high data unbalance, the models reach average precision 0.96-0.98 (0.75-0.85), recall 0.96-0.98 (0.74-0.85), and $F_1$-score 0.97-0.98 (0.79-0.83) on the non-hospitalized (or hospitalized) class. Performances …
This paper introduces X-SHAP, a model-agnostic method that assesses multiplicative contributions of variables for both local and global predictions. This method theoretically and operationally extends the so-called additive SHAP approach. It proves useful underlying multiplicative interactions of factors, typically arising in sectors where Generalized Linear Models are traditionally used, such as in insurance or biology. We test the method on various datasets and propose a set of techniques based on individual X-SHAP contributions to build aggregated multiplicative contributions and to capture…
ShaTS: A Shapley-based Explainability Method for Time Series Artificial Intelligence Models applied to Anomaly Detection in Industrial Internet of Things
Departamento de Ingeniería y Tecnología de Computadores, University of Murcia（计算机工程与技术部门，穆尔西亚大学）
Adobe（奥多比公司）；University of Tennessee（田纳西大学）；Harvard Business School（哈佛商学院）；Harvard University（哈佛大学）；Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学博德研究所）；Harvard Data Science Initiative（哈佛数据科学研究院）
As post hoc explanations are increasingly used to understand the behavior of graph neural networks (GNNs), it becomes crucial to evaluate the quality and reliability of GNN explanations. However, assessing the quality of GNN explanations is challenging as existing graph datasets have no or unreliable ground-truth explanations for a given task. Here, we introduce a synthetic graph data generator, ShapeGGen, which can generate a variety of benchmark datasets (e.g., varying graph sizes, degree distributions, homophilic vs. heterophilic graphs) accompanied by ground-truth explanations. Further, th…
Utilizing Large Language Models for Machine Learning Explainability
Centre for Research \& Technology, Hellas School of Architecture Technology；Engineering, University of Brighton, BN2 4GJ, United Kingdom University of Nicosia, 2417 Cyprus
Explainability for Fault Detection System in Chemical Processes
Department of Information and Electronic Engineering, International Hellenic University, Greece（信息与电子工程系，国际希腊大学）；Department of Industrial Engineering and Management, International Hellenic University, Greece（工业工程与管理系，国际希腊大学）；Energy Resources Institute, Centre for Research and Technology Hellas, Greece（能源资源研究所，希腊研究中心与技术中心）
Texas A&M University（德克萨斯农工大学）；University of Science and Technology of China（中国科学技术大学）；West China Hospital（华西医院）
We consider the problem of explaining the predictions of graph neural networks (GNNs), which otherwise are considered as black boxes. Existing methods invariably focus on explaining the importance of graph nodes or edges but ignore the substructures of graphs, which are more intuitive and human-intelligible. In this work, we propose a novel method, known as SubgraphX, to explain GNNs by identifying important subgraphs. Given a trained GNN model and an input graph, our SubgraphX explains its predictions by efficiently exploring different subgraphs with Monte Carlo tree search. To make the tree …
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；University of South Carolina（南卡罗来纳大学）
The vast majority of research on explainability focuses on post-explainability rather than explainable modeling. Namely, an explanation model is derived to explain a complex black box model built with the sole purpose of achieving the highest performance possible. In part, this trend might be driven by the misconception that there is a trade-off between explainability and accuracy. Furthermore, the consequential work on Shapely values, grounded in game theory, has also contributed to a new wave of post-explainability research on better approximations for various machine learning models, includ…
Tencent AI Lab（腾讯人工智能实验室）；Sun Yat-sen University（中山大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhejiang University（浙江大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）；Tsinghua University（清华大学）；Zhejiang Lab（之江实验室）
Deep graph learning has achieved remarkable progresses in both business and scientific areas ranging from finance and e-commerce, to drug and advanced material discovery. Despite these progresses, how to ensure various deep graph learning algorithms behave in a socially responsible manner and meet regulatory compliance requirements becomes an emerging problem, especially in risk-sensitive domains. Trustworthy graph learning (TwGL) aims to solve the above problems from a technical viewpoint. In contrast to conventional graph learning research which mainly cares about model performance, TwGL con…
UFRGS（里约热内卢联邦大学）；City, University of London（伦敦城市大学）；University of Siena（锡耶纳大学）；Université Côte d’Azur（蔚蓝海岸大学）；Rice University（莱斯大学）
Neural-symbolic computing has now become the subject of interest of both academic and industry research laboratories. Graph Neural Networks (GNN) have been widely used in relational and symbolic domains, with widespread application of GNNs in combinatorial optimization, constraint satisfaction, relational reasoning and other scientific domains. The need for improved explainability, interpretability and trust of AI systems in general demands principled methodologies, as suggested by neural-symbolic computing. In this paper, we review the state-of-the-art on the use of GNNs as a model of neural-…
University of Waterloo（滑铁卢大学）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）；DarwinAI Corp.（达尔文人工智能公司）
There has been a significant surge of interest recently around the concept of explainable artificial intelligence (XAI), where the goal is to produce an interpretation for a decision made by a machine learning algorithm. Of particular interest is the interpretation of how deep neural networks make decisions, given the complexity and `black box' nature of such networks. Given the infancy of the field, there has been very limited exploration into the assessment of the performance of explainability methods, with most evaluations centered around subjective visual interpretation of the produced int…
FlowX: Towards Explainable Graph Neural Networks via Message Flows
Texas A&M University（德克萨斯A&M大学）；University of Science and Technology of China（中国科学技术大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；West China Hospital（四川大学华西医院）
Detecting Anomalies in Blockchain Transactions using Machine Learning Classifiers and Explainability Analysis
Premier University（总理大学）；United International University（联合国际大学）；Cyber Security Cooperative Research Centre（网络安全合作研究中心）；Edith Cowan University（埃迪斯科文大学）
The recent increase in the deployment of machine learning models in critical domains such as healthcare, criminal justice, and finance has highlighted the need for trustworthy methods that can explain these models to stakeholders. Feature importance methods (e.g. gain and SHAP) are among the most popular explainability methods used to address this need. For any explainability technique to be trustworthy and meaningful, it has to provide an explanation that is accurate and stable. Although the stability of local feature importance methods (explaining individual predictions) has been studied bef…
Probabilistic Robustness-driven Universal Adversarial Perturbations with Explainability against Deep Reinforcement Learning-based Intrusion Detection System
University of Warwick（华威大学）；Loughborough University（拉夫堡大学）；Swansea University（斯旺西大学）
