The wide-spread adoption of representation learning technologies in clinical decision making strongly emphasizes the need for characterizing model reliability and enabling rigorous introspection of model behavior. While the former need is often addressed by incorporating uncertainty quantification strategies, the latter challenge is addressed using a broad class of interpretability techniques. In this paper, we argue that these two objectives are not necessarily disparate and propose to utilize prediction calibration to meet both objectives. More specifically, our approach is comprised of a calibration-driven learning method, which is also used to design an interpretability technique based on counterfactual reasoning. Furthermore, we introduce \textit{reliability plots}, a holistic evaluation mechanism for model reliability. Using a lesion classification problem with dermoscopy images, we demonstrate the effectiveness of our approach and infer interesting insights about the model behavior.
Recent advances in deep learning have led to the development of models approaching the human level of accuracy. However, healthcare remains an area lacking in widespread adoption. The safety-critical nature of healthcare results in a natural reticence to put these black-box deep learning models into practice. This paper explores interpretable methods for a clinical decision support system called sleep staging, an essential step in diagnosing sleep disorders. Clinical sleep staging is an arduous process requiring manual annotation for each 30s of sleep using physiological signals such as electr…
University of Southern California（南加州大学）；Children’s Hospital Los Angeles（洛杉矶儿童医院）
Exponential growth in Electronic Healthcare Records (EHR) has resulted in new opportunities and urgent needs for discovery of meaningful data-driven representations and patterns of diseases in Computational Phenotyping research. Deep Learning models have shown superior performance for robust prediction in computational phenotyping tasks, but suffer from the issue of model interpretability which is crucial for clinicians involved in decision-making. In this paper, we introduce a novel knowledge-distillation approach called Interpretable Mimic Learning, to learn interpretable phenotype features …
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）
Physiological signals are high-dimensional time series of great practical values in medical and healthcare applications. However, previous works on its classification fail to obtain promising results due to the intractable data characteristics and the severe label sparsity issues. In this paper, we try to address these challenges by proposing a more effective and interpretable scheme tailored for the physiological signal classification task. Specifically, we exploit the time series shapelets to extract prominent local patterns and perform interpretable sequence discretization to distill the wh…
Prototype-Based Learning for Healthcare: A Demonstration of Interpretable AI
NEC Laboratories Europe GmbH（NEC欧洲实验室）；NEC Solution Innovators, Ltd.（NEC解决方案创新者有限公司）；Kurashiki Central Hospital Preventive Healthcare Plaza（Kurashiki中央医院预防保健广场）；Kurashiki Central Hospital（Kurashiki中央医院）
University of Maribor（马里博尔大学）；Harvard University（哈佛大学）；KU Leuven（鲁汶大学）
There is a need of ensuring machine learning models that are interpretable. Higher interpretability of the model means easier comprehension and explanation of future predictions for end-users. Further, interpretable machine learning models allow healthcare experts to make reasonable and data-driven decisions to provide personalized decisions that can ultimately lead to higher quality of service in healthcare. Generally, we can classify interpretability approaches in two groups where the first focuses on personalized interpretation (local interpretability) while the second summarizes prediction…
Tensor factorization has been proved as an efficient unsupervised learning approach for health data analysis, especially for computational phenotyping, where the high-dimensional Electronic Health Records (EHRs) with patients' history of medical procedures, medications, diagnosis, lab tests, etc., are converted to meaningful and interpretable medical concepts. Federated tensor factorization distributes the tensor computation to multiple workers under the coordination of a central server, which enables jointly learning the phenotypes across multiple hospitals while preserving the privacy of the…
Deep Learning based models are currently dominating most state-of-the-art solutions for disease prediction. Existing works employ RNNs along with multiple levels of attention mechanisms to provide interpretability. These deep learning models, with trainable parameters running into millions, require huge amounts of compute and data to train and deploy. These requirements are sometimes so huge that they render usage of such models as unfeasible. We address these challenges by developing a simpler yet interpretable non-deep learning based model for application to EHR data. We model and showcase o…
Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework
Grado Department of Industrial & Systems Engineering, Virginia Tech（弗吉尼亚理工大学格拉多工业与系统工程系）；Institute for Intelligent Systems Research and Innovation (IISRI), Deakin University（迪肯大学智能系统研究与创新研究所）；School of Mathematics, Physics, and Computing, University of Southern Queensland（南昆士兰大学数学、物理与计算学院）；Center for Health Research, University of Southern Queensland（南昆士兰大学健康研究中心）
From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?
School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）；Harvard Medical School（哈佛医学院）；Boston Children’s Hospital（波士顿儿童医院）；University of Southampton（南安普顿大学）；Athena Research Centre（雅典娜研究中心）
Technical University of Munich（慕尼黑工业大学）；Ca’ Foscari University of Venice（威尼斯大学）；USI University of Lugano（瑞士意大利语区大学（卢加诺大学））；NVIDIA（英伟达）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Oxford（牛津大学）
Graphs are a powerful tool for representing and analyzing unstructured, non-Euclidean data ubiquitous in the healthcare domain. Two prominent examples are molecule property prediction and brain connectome analysis. Importantly, recent works have shown that considering relationships between input data samples have a positive regularizing effect for the downstream task in healthcare applications. These relationships are naturally modeled by a (possibly unknown) graph structure between input samples. In this work, we propose Graph-in-Graph (GiG), a neural network architecture for protein classifi…
Washington University in St. Louis（圣路易斯华盛顿大学）；Washington University School of Medicine（华盛顿大学医学院）
We investigate molecular mechanisms of resistant or sensitive response of cancer drug combination therapies in an inductive and interpretable manner. Though deep learning algorithms are widely used in the drug synergy prediction problem, it is still an open problem to formulate the prediction model with biological meaning to investigate the mysterious mechanisms of synergy (MoS) for the human-AI collaboration in healthcare systems. To address the challenges, we propose a deep graph neural network, IDSP (Interpretable Deep Signaling Pathways), to incorporate the gene-gene as well as gene-drug r…
Georgia Institute of Technology（佐治亚理工学院）；Sutter Health（萨特健康）
Accuracy and interpretability are two dominant features of successful predictive models. Typically, a choice must be made in favor of complex black box models such as recurrent neural networks (RNN) for accuracy versus less accurate but more interpretable traditional models such as logistic regression. This tradeoff poses challenges in medicine where both accuracy and interpretability are important. We addressed this challenge by developing the REverse Time AttentIoN model (RETAIN) for application to Electronic Health Records (EHR) data. RETAIN achieves high accuracy while remaining clinically…
Selecting Interpretability Techniques for Healthcare Machine Learning models
University of Antioquia UdeA（安蒂奥基亚大学）；Universidad de Antioquia UdeA（安蒂奥基亚大学）；University of Antioquia（安蒂奥基亚大学）；Hospital Alma Máter de Antioquia（安蒂奥基亚母校医院）；Arcada University of Applied Sciences（阿卡达应用科学大学）
We address the problem of calibrating prediction confidence for output entities of interest in natural language processing (NLP) applications. It is important that NLP applications such as named entity recognition and question answering produce calibrated confidence scores for their predictions, especially if the system is to be deployed in a safety-critical domain such as healthcare. However, the output space of such structured prediction models is often too large to adapt binary or multi-class calibration methods directly. In this study, we propose a general calibration scheme for output ent…
Retrieval-Augmented Interpretable Learning: Towards Task-Specific Zero-Shot Models in Healthcare
Carnegie Mellon University（卡内基梅隆大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；GenBio AI（基因生物人工智能公司）；Intel（英特尔公司）
National University of Singapore（新加坡国立大学）；National University Health System（新加坡国立大学医疗系统）
In high stakes applications such as healthcare and finance analytics, the interpretability of predictive models is required and necessary for domain practitioners to trust the predictions. Traditional machine learning models, e.g., logistic regression (LR), are easy to interpret in nature. However, many of these models aggregate time-series data without considering the temporal correlations and variations. Therefore, their performance cannot match up to recurrent neural network (RNN) based models, which are nonetheless difficult to interpret. In this paper, we propose a general framework TRACE…
IBM Research Zürich（IBM苏黎世研究）；ETH Zürich（苏黎世联邦理工学院）
Being able to interpret, or explain, the predictions made by a machine learning model is of fundamental importance. This is especially true when there is interest in deploying data-driven models to make high-stakes decisions, e.g. in healthcare. While recent years have seen an increasing interest in interpretable machine learning research, this field is currently lacking an agreed-upon definition of interpretability, and some researchers have called for a more active conversation towards a rigorous approach to interpretability. Joining this conversation, we claim in this paper that the difficu…
ETH Zürich（苏黎世联邦理工学院）；Yandex（俄罗斯搜索引擎公司）；UC Irvine（加州大学欧文分校）
Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can outperform classical data imputation methods in this domain. However, naive applications of deep learning fall short in giving reliable confidence estimates and lack interpretability. We propose a new deep sequential latent variable model for dimensionality reduction and data imputation. Our modeling assumption is simple and interpretable: the high dimensional time series has a l…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Rochester（罗切斯特大学）；Shanghai Maritime University（上海海事大学）；University of Rochester Medical Center（罗切斯特大学医学中心）
Causal inference and model interpretability research are gaining increasing attention, especially in the domains of healthcare and bioinformatics. Despite recent successes in this field, decorrelating features under nonlinear environments with human interpretable representations has not been adequately investigated. To address this issue, we introduce a novel method with a variable decorrelation regularizer to handle both linear and nonlinear confounding. Moreover, we employ association rules as new representations using association rule mining based on the original features to further proxima…
A machine learning framework for interpretable predictions in patient pathways: The case of predicting ICU admission for patients with symptoms of sepsis
Technische Hochschule Nürnberg Georg Simon Ohm（纽伦堡乔治·西蒙·欧姆应用技术大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；University of Regensburg（雷根斯堡大学）；Leipzig University（莱比锡大学）
Dept. of Informatics, Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学信息学系）
The use of machine learning rapidly increases in high-risk scenarios where decisions are required, for example in healthcare or industrial monitoring equipment. In crucial situations, a model that can offer meaningful explanations of its decision-making is essential. In industrial facilities, the equipment's well-timed maintenance is vital to ensure continuous operation to prevent money loss. Using machine learning, predictive and prescriptive maintenance attempt to anticipate and prevent eventual system failures. This paper introduces a visualisation tool incorporating interpretations to disp…
Florida State University（佛罗里达州立大学）；Tallahassee Memorial Hospital（塔拉哈西纪念医院）
Deep neural networks have achieved remarkable success in various challenging tasks. However, the black-box nature of such networks is not acceptable to critical applications, such as healthcare. In particular, the existence of adversarial examples and their overgeneralization to irrelevant, out-of-distribution inputs with high confidence makes it difficult, if not impossible, to explain decisions by such networks. In this paper, we analyze the underlying mechanism of generalization of deep neural networks and propose an ($n$, $k$) consensus algorithm which is insensitive to adversarial example…
An Interpretable Deep-Learning Framework for Predicting Hospital Readmissions From Electronic Health Records
Department of Electronics, Information and Bioengineering（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；Lero – the Science Foundation Ireland Research Centre for Software（爱尔兰科学基金会软件研究中心）；University of Limerick（利默ick大学）
Interpretable Survival Analysis for Heart Failure Risk Prediction
Stanford University（斯坦福大学）；VA Palo Alto Health Care System（帕洛阿尔托退伍军人医疗保健系统）；Kaiser Permanente Northern California Division of Research（凯撒医疗北加州研究部）；Stanford Medicine（斯坦福医学院）
Georgia Institute of Technology（佐治亚理工学院）；California Institute of Technology（加州理工学院）；Sutter Health（萨特健康）
In application domains such as healthcare, we want accurate predictive models that are also causally interpretable. In pursuit of such models, we propose a causal regularizer to steer predictive models towards causally-interpretable solutions and theoretically study its properties. In a large-scale analysis of Electronic Health Records (EHR), our causally-regularized model outperforms its L1-regularized counterpart in causal accuracy and is competitive in predictive performance. We perform non-linear causality analysis by causally regularizing a special neural network architecture. We also sho…
Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning
University of Cambridge（剑桥大学）；Bilkent University（比尔肯大学）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；The Alan Turing Institute（艾伦·图灵研究所）；UCLA（加州大学洛杉矶分校）
Navigating Healthcare Insights: A Birds Eye View of Explainability with Knowledge Graphs
Hajim School of Engineering and Applied Sciences, University of Rochester（罗切斯特大学哈吉米工程与应用科学学院）；College of Engineering and Applied Sciences, University at Albany, State University of New York（纽约州立大学奥尔巴尼分校工程与应用科学学院）；Deloitte LLC（德勤有限责任公司）
Auckland University of Technology（奥克兰理工大学）；Universidad Tecnológica de Bolívar（玻利瓦尔理工大学）
To improve the performance of Intensive Care Units (ICUs), the field of bio-statistics has developed scores which try to predict the likelihood of negative outcomes. These help evaluate the effectiveness of treatments and clinical practice, and also help to identify patients with unexpected outcomes. However, they have been shown by several studies to offer sub-optimal performance. Alternatively, Deep Learning offers state of the art capabilities in certain prediction tasks and research suggests deep neural networks are able to outperform traditional techniques. Nevertheless, a main impediment…
Tsinghua National Lab for Information Science and Technology（清华大学信息科学与技术国家实验室）；State Key Lab of Intelligent Technology and Systems（智能技术与系统国家重点实验室）；Center for Bio-Inspired Computing Research（生物启发计算研究中心）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
Sometimes it is not enough for a DNN to produce an outcome. For example, in applications such as healthcare, users need to understand the rationale of the decisions. Therefore, it is imperative to develop algorithms to learn models with good interpretability (Doshi-Velez 2017). An important factor that leads to the lack of interpretability of DNNs is the ambiguity of neurons, where a neuron may fire for various unrelated concepts. This work aims to increase the interpretability of DNNs on the whole image space by reducing the ambiguity of neurons. In this paper, we make the following contribut…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Hospital of Sickkids（病童医院）；Microsoft Research（微软研究院）
Deployment of machine learning models in real high-risk settings (e.g. healthcare) often depends not only on the model's accuracy but also on its fairness, robustness, and interpretability. Generalized Additive Models (GAMs) are a class of interpretable models with a long history of use in these high-risk domains, but they lack desirable features of deep learning such as differentiability and scalability. In this work, we propose a neural GAM (NODE-GAM) and neural GA$^2$M (NODE-GA$^2$M) that scale well and perform better than other GAMs on large datasets, while remaining interpretable compared…
BASIS Chandler（贝赛思钱德勒学校）；SRI International（斯坦福国际研究院）；Arizona State University（亚利桑那州立大学）
Deep neural networks have increasingly been used as an auxiliary tool in healthcare applications, due to their ability to improve performance of several diagnosis tasks. However, these methods are not widely adopted in clinical settings due to the practical limitations in the reliability, generalizability, and interpretability of deep learning based systems. As a result, methods have been developed that impose additional constraints during network training to gain more control as well as improve interpretabilty, facilitating their acceptance in healthcare community. In this work, we investigat…
Who Became Financially Vulnerable After COVID-19? A Population-Level Machine Learning Analysis Using MEPS Data
Hood College（胡德学院）；Georgetown University（乔治敦大学）；University of Benin（贝宁大学）；NIST University（NIST大学）；AI CoLab: MedStar–Georgetown Collaborative Center for Artificial Intelligence in Healthcare Research and Education（人工智能合作实验室：梅斯达星 - 乔治敦医疗保健研究与教育人工智能协作中心）；MedStar Health Research Institute（梅斯达星健康研究所）；Albany State University（奥尔巴尼州立大学）
CNRS-LIMSI（法国国家科学研究中心-计算机科学与机械信息实验室）；Université Paris-Saclay（巴黎-萨克雷大学）；Samovar, CNRS, Télécom SudParis, Institut Polytechnique de Paris（萨莫瓦尔实验室，法国国家科学研究中心，南巴黎电信学院，巴黎综合理工学院）；Université Paris 8（巴黎第八大学）
The adoption of deep learning in healthcare is hindered by their "black box" nature. In this paper, we explore the RETAIN architecture for the task of glusose forecasting for diabetic people. By using a two-level attention mechanism, the recurrent-neural-network-based RETAIN model is interpretable. We evaluate the RETAIN model on the type-2 IDIAB and the type-1 OhioT1DM datasets by comparing its statistical and clinical performances against two deep models and three models based on decision trees. We show that the RETAIN model offers a very good compromise between accuracy and interpretability…
The Managed Care system within Medicaid (US Healthcare) uses Request For Proposals (RFP) to award contracts for various healthcare and related services. RFP responses are very detailed documents (hundreds of pages) submitted by competing organisations to win contracts. Subject matter expertise and domain knowledge play an important role in preparing RFP responses along with analysis of historical submissions. Automated analysis of these responses through Natural Language Processing (NLP) systems can reduce time and effort needed to explore historical responses, and assisting in writing better …
University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；Decyphir, Inc（Decyphir公司）
Cyber-physical system applications such as autonomous vehicles, wearable devices, and avionic systems generate a large volume of time-series data. Designers often look for tools to help classify and categorize the data. Traditional machine learning techniques for time-series data offer several solutions to solve these problems; however, the artifacts trained by these algorithms often lack interpretability. On the other hand, temporal logics, such as Signal Temporal Logic (STL) have been successfully used in the formal methods community as specifications of time-series behaviors. In this work, …
Emory University（埃默里大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Princeton University（普林斯顿大学）；Lehigh University（理海大学）
Interpretable brain network models for disease prediction are of great value for the advancement of neuroscience. GNNs are promising to model complicated network data, but they are prone to overfitting and suffer from poor interpretability, which prevents their usage in decision-critical scenarios like healthcare. To bridge this gap, we propose BrainNNExplainer, an interpretable GNN framework for brain network analysis. It is mainly composed of two jointly learned modules: a backbone prediction model that is specifically designed for brain networks and an explanation generator that highlights …
Multi-Stage Prototype Learning for Interpretable Time Series Classification
Department of Neurology, University of California, San Francisco（加州大学旧金山分校神经科部门）；Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco（加州大学旧金山分校生物工程与治疗科学部门）；UCSF Weill Institute for Neurosciences（UCSF Weill神经科学研究所）
The increasing availability of large collections of electronic health record (EHR) data and unprecedented technical advances in deep learning (DL) have sparked a surge of research interest in developing DL based clinical decision support systems for diagnosis, prognosis, and treatment. Despite the recognition of the value of deep learning in healthcare, impediments to further adoption in real healthcare settings remain due to the black-box nature of DL. Therefore, there is an emerging need for interpretable DL, which allows end users to evaluate the model decision making to know whether to acc…
