Artificial Intelligence has emerged as a useful aid in numerous clinical applications for diagnosis and treatment decisions. Deep neural networks have shown same or better performance than clinicians in many tasks owing to the rapid increase in the available data and computational power. In order to conform to the principles of trustworthy AI, it is essential that the AI system be transparent, robust, fair and ensure accountability. Current deep neural solutions are referred to as black-boxes due to a lack of understanding of the specifics concerning the decision making process. Therefore, there is a need to ensure interpretability of deep neural networks before they can be incorporated in the routine clinical workflow. In this narrative review, we utilized systematic keyword searches and domain expertise to identify nine different types of interpretability methods that have been used for understanding deep learning models for medical image analysis applications based on the type of generated explanations and technical similarities. Furthermore, we report the progress made towards evaluating the explanations produced by various interpretability methods. Finally we discuss limitations, provide guidelines for using interpretability methods and future directions concerning the interpretability of deep neural networks for medical imaging analysis.
Transparency in Machine Learning (ML), attempts to reveal the working mechanisms of complex models. Transparent ML promises to advance human factors engineering goals of human-centered AI in the target users. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e. a relationship between algorithm and user; as a result, iterative prototyping and evaluation with users is critical to attaining adequate solutions that afford transparency. However, following human-centered design principles in healthcare and medical image analysis is challeng…
Deep learning methods, in particular convolutional neural networks, have emerged as a powerful tool in medical image computing tasks. While these complex models provide excellent performance, their black-box nature may hinder real-world adoption in high-stakes decision-making. In this paper, we propose an interactive system to take advantage of state-of-the-art interpretability techniques to assist radiologists with breast cancer screening. Our system integrates a deep learning model into the radiologists' workflow and provides novel interactions to promote understanding of the model's decisio…
Towards Multi-dimensional Explanation Alignment for Medical Classification
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；HKUST(GZ)（香港科技大学（广州））；Tongji University（同济大学）；Ant Group（蚂蚁集团）；The University of Auckland（奥克兰大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心）
As AI-based medical devices are becoming more common in imaging fields like radiology and histology, interpretability of the underlying predictive models is crucial to expand their use in clinical practice. Existing heatmap-based interpretability methods such as GradCAM only highlight the location of predictive features but do not explain how they contribute to the prediction. In this paper, we propose a new interpretability method that can be used to understand the predictions of any black-box model on images, by showing how the input image would be modified in order to produce different pred…
Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this paper, we propose a novel interpretation method tailored to histological Whole Slide Image (WSI) processing. A Deep Neural Network (DNN), inspired by Bag-of-Features models is equipped with a Multiple Instance Learning (MIL) branch and trained with weak supervision for WSI classification. MIL avoids label ambiguity and enhances our model's expressive power without guiding its attention. We utilize a fine-grained logit heatmap of the models activations to interpret its decision-making process. The proposed method is quantitatively and qualitatively evaluated on two challenging histology…
Peter Munk Cardiac Center（彼得·芒克心脏中心）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Waterloo（滑铁卢大学）
Medical image segmentation is a difficult but important task for many clinical operations such as cardiac bi-ventricular volume estimation. More recently, there has been a shift to utilizing deep learning and fully convolutional neural networks (CNNs) to perform image segmentation that has yielded state-of-the-art results in many public benchmark datasets. Despite the progress of deep learning in medical image segmentation, standard CNNs are still not fully adopted in clinical settings as they lack robustness and interpretability. Shapes are generally more meaningful features than solely textu…
Athinoula A. Martinos Center for Biomedical Imaging（阿西娜·A·马蒂诺斯生物医学影像中心）；Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）
Deep learning has the potential to automate many clinically useful tasks in medical imaging. However translation of deep learning into clinical practice has been hindered by issues such as lack of the transparency and interpretability in these "black box" algorithms compared to traditional statistical methods. Specifically, many clinical deep learning models lack rigorous and robust techniques for conveying certainty (or lack thereof) in their predictions -- ultimately limiting their appeal for extensive use in medical decision-making. Furthermore, numerous demonstrations of algorithmic bias h…
MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification
Department of Diagnostic and Interventional Radiology of the University Hospital RWTH Aachen（鲁尔大学亚琛医院诊断与介入放射科）；Else Kroener Fresenius Center for Digital Health, Technical University Dresden（德累斯顿技术大学埃勒克罗纳弗雷斯内尔数字健康中心）；Department of Medicine III, University Hospital RWTH Aachen（鲁尔大学亚琛医院第三医学部）；Pathology & Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds（利兹大学圣詹姆斯医院医学研究所以及病理学与数据科学）；Department of Medicine I, University Hospital Dresden（德累斯顿大学第一医学部）；Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg（海德堡大学国家肿瘤疾病中心（NCT）肿瘤医学部）
University of Porto（波尔图大学）；INESC TEC（葡萄牙系统与计算机工程研究所）
The use of Deep Learning in the medical field is hindered by the lack of interpretability. Case-based interpretability strategies can provide intuitive explanations for deep learning models' decisions, thus, enhancing trust. However, the resulting explanations threaten patient privacy, motivating the development of privacy-preserving methods compatible with the specifics of medical data. In this work, we analyze existing privacy-preserving methods and their respective capacity to anonymize medical data while preserving disease-related semantic features. We find that the PPRL-VGAN deep learning…
Deep learning (DL) methods where interpretability is intrinsically considered as part of the model are required to better understand the relationship of clinical and imaging-based attributes with DL outcomes, thus facilitating their use in the reasoning behind medical decisions. Latent space representations built with variational autoencoders (VAE) do not ensure individual control of data attributes. Attribute-based methods enforcing attribute disentanglement have been proposed in the literature for classical computer vision tasks in benchmark data. In this paper, we propose a VAE approach, th…
Manipal Institute of Technology（马尼帕尔理工学院）；Carnegie Mellon University（卡内基梅隆大学）；OpenMined；University of Pennsylvania（宾夕法尼亚大学）；IIT Jodhpur（印度理工学院焦特布尔分校）；University of California, San Diego（加州大学圣地亚哥分校）
Given the increase in the use of personal data for training Deep Neural Networks (DNNs) in tasks such as medical imaging and diagnosis, differentially private training of DNNs is surging in importance and there is a large body of work focusing on providing better privacy-utility trade-off. However, little attention is given to the interpretability of these models, and how the application of DP affects the quality of interpretations. We propose an extensive study into the effects of DP training on DNNs, especially on medical imaging applications, on the APTOS dataset.
Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder
Technical University of Munich（慕尼黑工业大学）；Klinikum rechts der Isar（右侧伊萨尔医院）；Institute for Artificial Intelligence and Computer Science in Medicine, Technical University of Munich（慕尼黑工业大学人工智能与医学计算机科学研究所）；Computer Aided Medical Procedures, Technical University of Munich（慕尼黑工业大学计算机辅助医疗程序研究所）；AI for Image-Guided Diagnosis and Therapy, Technical University of Munich（慕尼黑工业大学图像引导诊断与治疗人工智能研究所）
The practical application of deep neural networks are still limited by their lack of transparency. One of the efforts to provide explanation for decisions made by artificial intelligence (AI) is the use of saliency or heat maps highlighting relevant regions that contribute significantly to its prediction. A layer-wise amplitude filtering method was previously introduced to improve the quality of heatmaps, performing error corrections by noise-spike suppression. In this study, we generalize the layerwise error correction by considering any identifiable error and assuming there exists a groundtr…
Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
Helmholtz Pioneer Campus, Helmholtz Munich（亥姆霍兹先锋园区，亥姆霍兹慕尼黑中心）；Biozentrum, University of Basel（巴塞尔大学生物中心）；LMU University Hospital, LMU Munich（慕尼黑大学医院，慕尼黑大学）
Learning Concept-Driven Logical Rules for Interpretable and Generalizable Medical Image Classification
School of Data Science, Fudan University（复旦大学数据科学学院）；Dept. of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Dept. of Oncology, University of Cambridge（剑桥大学肿瘤学系）
Deep Learning has achieved state of the art performance in medical imaging. However, these methods for disease detection focus exclusively on improving the accuracy of classification or predictions without quantifying uncertainty in a decision. Knowing how much confidence there is in a computer-based medical diagnosis is essential for gaining clinicians trust in the technology and therefore improve treatment. Today, the 2019 Coronavirus (SARS-CoV-2) infections are a major healthcare challenge around the world. Detecting COVID-19 in X-ray images is crucial for diagnosis, assessment and treatmen…
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA（系统工程系，史蒂文斯理工学院）；Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan（高雄长庚纪念医院核医学部）；Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan（国立高雄大学统计研究所）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China（深圳先进技术研究院，中国科学院）
Interpretability of deep learning (DL) systems is gaining attention in medical imaging to increase experts' trust in the obtained predictions and facilitate their integration in clinical settings. We propose a deep visualization method to generate interpretability of DL classification tasks in medical imaging by means of visual evidence augmentation. The proposed method iteratively unveils abnormalities based on the prediction of a classifier trained only with image-level labels. For each image, initial visual evidence of the prediction is extracted with a given visual attribution technique. T…
Concept Complement Bottleneck Model for Interpretable Medical Image Diagnosis
Hong Kong University of Science and Technology（香港科技大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Chemical and Biological Engineering（化学与生物工程系）；Division of Life Science（生命科学部）
Generalizable and Explainable Deep Learning for Medical Image Computing: An Overview
School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；École de Technologie Supérieure (ETS)（高等技术学院）；College of Computer Science and Engineering, Taibah University（塔伊布大学计算机科学与工程学院）；College of Computer Science and Engineering, Jeddah University（吉达大学计算机科学与工程学院）
Graph neural networks (GNNs) are highly effective on a variety of graph-related tasks; however, they lack interpretability and transparency. Current explainability approaches are typically local and treat GNNs as black-boxes. They do not look inside the model, inhibiting human trust in the model and explanations. Motivated by the ability of neurons to detect high-level semantic concepts in vision models, we perform a novel analysis on the behaviour of individual GNN neurons to answer questions about GNN interpretability, and propose new metrics for evaluating the interpretability of GNN neuron…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）
Deep learning as represented by the artificial deep neural networks (DNNs) has achieved great success in many important areas that deal with text, images, videos, graphs, and so on. However, the black-box nature of DNNs has become one of the primary obstacles for their wide acceptance in mission-critical applications such as medical diagnosis and therapy. Due to the huge potential of deep learning, interpreting neural networks has recently attracted much research attention. In this paper, based on our comprehensive taxonomy, we systematically review recent studies in understanding the mechanis…
Interpretable Medical Image Classification using Prototype Learning and Privileged Information
Experimental Radiology, University Hospital Ulm, Germany（实验放射学，乌尔姆大学医院，德国）；Department of Diagnostic and Interventional Radiology, University Hospital Ulm, Germany（诊断与介入放射学系，乌尔姆大学医院，德国）；i2SouI - Innovative Imaging in Surgical Oncology Ulm, University Hospital Ulm, Germany（i2SouI - 精准影像学在手术肿瘤学中的创新应用，乌尔姆大学医院，德国）
Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
Department of Engineering（工程系）；Research Unit of Artificial Intelligence and Computer Systems（人工智能与计算机系统研究单位）；Università Campus Bio-Medico of Roma（罗马大学生物医学校园）；Department of Computer, Control and Management Engineering（计算机、控制与管理工程系）；Sapienza University of Rome（罗马萨皮恩扎大学）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering（诊断与介入、辐射物理、生物医学工程系）；Umeå University（乌梅拉大学）；UniCamillus-Saint Camillus International University of Health Sciences（UniCamillus-圣卡米卢斯国际健康科学大学）
GFSR-Net: Guided Focus via Segment-Wise Relevance Network for Interpretable Deep Learning in Medical Imaging
Institute of Physical Chemistry (IPC) and Abbe Center of Photonics (ACP), Friedrich Schiller University Jena（耶拿弗里德里希·席勒大学物理化学研究所与阿贝光子学中心）；Leibniz Institute of Photonic Technology（莱布尼茨光子技术研究所）
NeRD: Neuro-Symbolic Rule Distillation for Efficient Ontology-Grounded Chain-of-Thought in Medical Image Diagnosis
Department of Data Science & AI, Faculty of Information Technology, Monash University（莫纳什大学信息技术学院数据科学与人工智能系）；AIM for Health Lab, Faculty of Information Technology, Monash University（莫纳什大学信息技术学院AIM健康实验室）；Faculty of Engineering, Monash University（莫纳什大学工程学院）；Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）
Using Computer Vision for Skin Disease Diagnosis in Bangladesh Enhancing Interpretability and Transparency in Deep Learning Models for Skin Cancer Classification
SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
Stockholm University（斯德哥尔摩大学）；Skatturinn (Iceland Revenue and Customs)（冰岛税务总局（冰岛税收和海关局））
Northeastern University（东北大学）；Brigham and Women’s Hospital（布莱根妇女医院）；University of Amsterdam（阿姆斯特丹大学）
Pretraining multimodal models on Electronic Health Records (EHRs) provides a means of learning representations that can transfer to downstream tasks with minimal supervision. Recent multimodal models induce soft local alignments between image regions and sentences. This is of particular interest in the medical domain, where alignments might highlight regions in an image relevant to specific phenomena described in free-text. While past work has suggested that attention "heatmaps" can be interpreted in this manner, there has been little evaluation of such alignments. We compare alignments from a…
As interpretability has been pointed out as the obstacle to the adoption of Deep Neural Networks (DNNs), there is an increasing interest in solving a transparency issue to guarantee the impressive performance. In this paper, we demonstrate the efficiency of recent attribution techniques to explain the diagnostic decision by visualizing the significant factors in the input image. By utilizing the characteristics of objectness that DNNs have learned, fully decomposing the network prediction visualizes clear localization of target lesion. To verify our work, we conduct our experiments on Chest X-…
Objective: We develop a computer-aided diagnosis (CAD) system using deep learning approaches for lesion detection and classification on whole-slide images (WSIs) with breast cancer. The deep features being distinguishing in classification from the convolutional neural networks (CNN) are demonstrated in this study to provide comprehensive interpretability for the proposed CAD system using pathological knowledge. Methods: In the experiment, a total of 186 slides of WSIs were collected and classified into three categories: Non-Carcinoma, Ductal Carcinoma in Situ (DCIS), and Invasive Ductal Carcin…
