Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the "opaque" nature of ML where the learnt model is not amenable to human scrutiny. Explainable AI (XAI) methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.
Navigating Healthcare Insights: A Birds Eye View of Explainability with Knowledge Graphs
Hajim School of Engineering and Applied Sciences, University of Rochester（罗切斯特大学哈吉米工程与应用科学学院）；College of Engineering and Applied Sciences, University at Albany, State University of New York（纽约州立大学奥尔巴尼分校工程与应用科学学院）；Deloitte LLC（德勤有限责任公司）
The University of Manchester（曼彻斯特大学）；CRUK Manchester Institute（英国癌症研究中心曼彻斯特研究所）；Idiap Research Institute（Idiap研究所）
This paper contributes with a pragmatic evaluation framework for explainable Machine Learning (ML) models for clinical decision support. The study revealed a more nuanced role for ML explanation models, when these are pragmatically embedded in the clinical context. Despite the general positive attitude of healthcare professionals (HCPs) towards explanations as a safety and trust mechanism, for a significant set of participants there were negative effects associated with confirmation bias, accentuating model over-reliance and increased effort to interact with the model. Also, contradicting one …
Amazon ML Solutions Lab（亚马逊机器学习解决方案实验室）；Merck & Co., Inc.（默沙东公司）
Recent advances in deep learning have led to interest in training deep learning models on longitudinal healthcare records to predict a range of medical events, with models demonstrating high predictive performance. Predictive performance is necessary but insufficient, however, with explanations and reasoning from models required to convince clinicians for sustained use. Rigorous evaluation of explainability is often missing, as comparisons between models (traditional versus deep) and various explainability methods have not been well-studied. Furthermore, ground truths needed to evaluate explai…
DarwinAI Corp.（DarwinAI公司）；University of Waterloo（滑铁卢大学）；Waterloo Artificial Intelligence Institute（滑铁卢人工智能研究所）
The COVID-19 pandemic continues to have a devastating global impact, and has placed a tremendous burden on struggling healthcare systems around the world. Given the limited resources, accurate patient triaging and care planning is critical in the fight against COVID-19, and one crucial task within care planning is determining if a patient should be admitted to a hospital's intensive care unit (ICU). Motivated by the need for transparent and trustworthy ICU admission clinical decision support, we introduce COVID-Net Clinical ICU, a neural network for ICU admission prediction based on patient cl…
Explainability matters: The effect of liability rules on the healthcare sector
Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）；Dipartimento di Giurisprudenza, Università di Pisa（比萨大学法学院）；Dirpolis Institute, Scuola Superiore Sant’Anna（圣安娜高等学院Dirpolis研究所）；Department of Mathematics (courtesy), National University of Singapore（新加坡国立大学数学系）
FairMed-XGB: A Bayesian-Optimised Multi-Metric Framework with Explainability for Demographic Equity in Critical Healthcare Data
School of Computer Engineering, Kalinga Institute of Industrial Technology（计算机工程学院，卡林加工业技术学院）；Bio-AI Lab, UiT The Arctic University of Norway（生物-人工智能实验室，北欧大学（UiT））
MedISure: Towards Assuring Machine Learning-based Medical Image Classifiers using Mixup Boundary Analysis
NHS England (National Health Service)（英格兰国民保健制度）；Information Technology University of the Punjab（旁遮普信息技术大学）；Qatar University（卡塔尔大学）；Birmingham City University (BCU)（伯明翰城市大学）
Pandit Deendayal Petroleum University（潘迪特·迪恩达亚尔石油大学）；Confirm SFI Centre for Smart Manufacturing（Confirm SFI智能制造中心）；Data Science Institute（数据科学研究所）；National University of Ireland Galway（爱尔兰国立大学戈尔韦分校）
With the increasing availability of structured and unstructured data and the swift progress of analytical techniques, Artificial Intelligence (AI) is bringing a revolution to the healthcare industry. With the increasingly indispensable role of AI in healthcare, there are growing concerns over the lack of transparency and explainability in addition to potential bias encountered by predictions of the model. This is where Explainable Artificial Intelligence (XAI) comes into the picture. XAI increases the trust placed in an AI system by medical practitioners as well as AI researchers, and thus, ev…
Self-explaining Neural Network with Concept-based Explanations for ICU Mortality Prediction
Department of Computer Science and Engineering, Washington University in St. Louis, St. Louis, MO, USA（计算机科学与工程系，华盛顿大学圣路易斯分校）；Institute for Informatics, Washington University School of Medicine, St. Louis, MO, USA（信息学院，华盛顿大学医学学院）；Department of Anaesthesiology, Washington University School of Medicine, St. Louis, MO, USA（麻醉学系，华盛顿大学医学学院）；Department of Pulmonary Critical Care and Medicine, Washington University School of Medicine, St. Louis, MO, USA（呼吸科重症医学系，华盛顿大学医学学院）
Massachusetts Institute of Technology（麻省理工学院）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Unity Health Toronto（多伦多团结健康医院）
Machine learning models in safety-critical settings like healthcare are often blackboxes: they contain a large number of parameters which are not transparent to users. Post-hoc explainability methods where a simple, human-interpretable model imitates the behavior of these blackbox models are often proposed to help users trust model predictions. In this work, we audit the quality of such explanations for different protected subgroups using real data from four settings in finance, healthcare, college admissions, and the US justice system. Across two different blackbox model architectures and fou…
The use of machine learning (ML) models in decision-making contexts, particularly those used in high-stakes decision-making, are fraught with issue and peril since a person - not a machine - must ultimately be held accountable for the consequences of the decisions made using such systems. Machine learning explainability (MLX) promises to provide decision-makers with prediction-specific rationale, assuring them that the model-elicited predictions are made for the right reasons and are thus reliable. Few works explicitly consider this key human-in-the-loop (HITL) component, however. In this work…
Signature Activation: A Sparse Signal View for Holistic Saliency
Harvard University（哈佛大学）；Broad Institute of Massachusetts Institute of Technology and Harvard（麻省理工学院与哈佛大学博德研究所）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Defence Science and Technology Laboratory（国防科学与技术实验室）；University of York（约克大学）
Machine learning has evolved into an enabling technology for a wide range of highly successful applications. The potential for this success to continue and accelerate has placed machine learning (ML) at the top of research, economic and political agendas. Such unprecedented interest is fuelled by a vision of ML applicability extending to healthcare, transportation, defence and other domains of great societal importance. Achieving this vision requires the use of ML in safety-critical applications that demand levels of assurance beyond those needed for current ML applications. Our paper provides…
NHS Digital（NHS数字部）；Kheiron Medical Technologies（凯龙医疗技术公司）；Health Navigator（健康导航公司）；University of York（约克大学）
In recent years, the number of machine learning (ML) technologies gaining regulatory approval for healthcare has increased significantly allowing them to be placed on the market. However, the regulatory frameworks applied to them were originally devised for traditional software, which has largely rule-based behaviour, compared to the data-driven and learnt behaviour of ML. As the frameworks are in the process of reformation, there is a need to proactively assure the safety of ML to prevent patient safety being compromised. The Assurance of Machine Learning for use in Autonomous Systems (AMLAS)…
We propose a novel approach that integrates machine learning into compartmental disease modeling to predict the progression of COVID-19. Our model is explainable by design as it explicitly shows how different compartments evolve and it uses interpretable encoders to incorporate covariates and improve performance. Explainability is valuable to ensure that the model's forecasts are credible to epidemiologists and to instill confidence in end-users such as policy makers and healthcare institutions. Our model can be applied at different geographic resolutions, and here we demonstrate it for states…
Although "black box" models such as Artificial Neural Networks, Support Vector Machines, and Ensemble Approaches continue to show superior performance in many disciplines, their adoption in the sensitive disciplines (e.g., finance, healthcare) is questionable due to the lack of interpretability and explainability of the model. In fact, future adoption of "black box" models is difficult because of the recent rule of "right of explanation" by the European Union where a user can ask for an explanation behind an algorithmic decision, and the newly proposed bill by the US government, the "Algorithm…
Explainability has been a goal for Artificial Intelligence (AI) systems since their conception, with the need for explainability growing as more complex AI models are increasingly used in critical, high-stakes settings such as healthcare. Explanations have often added to an AI system in a non-principled, post-hoc manner. With greater adoption of these systems and emphasis on user-centric explainability, there is a need for a structured representation that treats explainability as a primary consideration, mapping end user needs to specific explanation types and the system's AI capabilities. We …
Digital Reasoning / Smarsh, Inc（Digital Reasoning / Smarsh公司）
The healthcare domain is one of the most exciting application areas for machine learning, but a lack of model transparency contributes to a lag in adoption within the industry. In this work, we explore the current art of explainability and interpretability within a case study in clinical text classification, using a task of mortality prediction within MIMIC-III clinical notes. We demonstrate various visualization techniques for fully interpretable methods as well as model-agnostic post hoc attributions, and we provide a generalized method for evaluating the quality of explanations using infide…
Fraunhofer Institute for Optronics, System Technologies, and Image Exploitation IOSB（弗劳恩霍夫光电、系统技术与图像利用研究所IOSB）；University of Stuttgart（斯图加特大学）；Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所IPA）
Predictions obtained by, e.g., artificial neural networks have a high accuracy but humans often perceive the models as black boxes. Insights about the decision making are mostly opaque for humans. Particularly understanding the decision making in highly sensitive areas such as healthcare or fifinance, is of paramount importance. The decision-making behind the black boxes requires it to be more transparent, accountable, and understandable for humans. This survey paper provides essential definitions, an overview of the different principles and methodologies of explainable Supervised Machine Lear…
We are witnessing the emergence of an AI economy and society where AI technologies are increasingly impacting health care, business, transportation and many aspects of everyday life. Many successes have been reported where AI systems even surpassed the accuracy of human experts. However, AI systems may produce errors, can exhibit bias, may be sensitive to noise in the data, and often lack technical and judicial transparency resulting in reduction in trust and challenges in their adoption. These recent shortcomings and concerns have been documented in scientific but also in general press such a…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；The Hospital for Sick Children（病童医院）
Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care…
Harvard University（哈佛大学）；UC Irvine（加州大学欧文分校）；University of Chicago（芝加哥大学）；AI2（艾伦人工智能研究所）
As practitioners increasingly deploy machine learning models in critical domains such as health care, finance, and policy, it becomes vital to ensure that domain experts function effectively alongside these models. Explainability is one way to bridge the gap between human decision-makers and machine learning models. However, most of the existing work on explainability focuses on one-off, static explanations like feature importances or rule lists. These sorts of explanations may not be sufficient for many use cases that require dynamic, continuous discovery from stakeholders. In the literature,…
Amsterdam University Medical Center, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Institute for Logic, Language, and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Pacmed；Amsterdam Business School, University of Amsterdam（阿姆斯特丹大学阿姆斯特丹商学院）
The recent spike in certified Artificial Intelligence (AI) tools for healthcare has renewed the debate around adoption of this technology. One thread of such debate concerns Explainable AI (XAI) and its promise to render AI devices more transparent and trustworthy. A few voices active in the medical AI space have expressed concerns on the reliability of Explainable AI techniques and especially feature attribution methods, questioning their use and inclusion in guidelines and standards. Despite valid concerns, we argue that existing criticism on the viability of post-hoc local explainability me…
REX: Causal discovery based on machine learning and explainability techniques
BBVA Madrid, Spain（西班牙BBVA马德里）；Instituto de Ciencia de Datos e Inteligencia Artificil (DATAI)（数据与人工智能研究所）；University of Navarra（纳瓦拉大学）；Tecnun, University of Navarra（纳瓦拉大学技术学院）
University of Waterloo（滑铁卢大学）；National Research Council Canada（加拿大国家研究委员会）
Building AI models with trustworthiness is important especially in regulated areas such as healthcare. In tackling COVID-19, previous work uses convolutional neural networks as the backbone architecture, which has shown to be prone to over-caution and overconfidence in making decisions, rendering them less trustworthy -- a crucial flaw in the context of medical imaging. In this study, we propose a feature learning approach using Vision Transformers, which use an attention-based mechanism, and examine the representation learning capability of Transformers as a new backbone architecture for medi…
Machine Learning (ML) models are increasingly used to make critical decisions in real-world applications, yet they have become more complex, making them harder to understand. To this end, researchers have proposed several techniques to explain model predictions. However, practitioners struggle to use these explainability techniques because they often do not know which one to choose and how to interpret the results of the explanations. In this work, we address these challenges by introducing TalkToModel: an interactive dialogue system for explaining machine learning models through conversations…
From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?
School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）；Harvard Medical School（哈佛医学院）；Boston Children’s Hospital（波士顿儿童医院）；University of Southampton（南安普顿大学）；Athena Research Centre（雅典娜研究中心）
University of Washington（华盛顿大学）；Arthur AI（亚瑟人工智能公司）
Machine learning plays a role in many deployed decision systems, often in ways that are difficult or impossible to understand by human stakeholders. Explaining, in a human-understandable way, the relationship between the input and output of machine learning models is essential to the development of trustworthy machine learning based systems. A burgeoning body of research seeks to define the goals and methods of explainability in machine learning. In this paper, we seek to review and categorize research on counterfactual explanations, a specific class of explanation that provides a link between…
Unifying Interpretability and Explainability for Alzheimer's Disease Progression Prediction
Kansas State University（堪萨斯州立大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Kansas Medical Center（堪萨斯大学医学中心）；Alzheimer’s Disease Research Center（阿尔茨海默病研究中心）
