Deep learning models for medical imaging often exhibit overconfidence, creating safety risks in ambiguous diagnostic scenarios. While Conformal Prediction (CP) provides distribution-free statistical guarantees, standard methods such as Regularized Adaptive Prediction Sets (RAPS) optimize for average efficiency and can mask severe failures on difficult inputs.
We propose an Adaptive Lambda Criterion for RAPS that minimizes the worst-case coverage violation across prediction set size strata. On OrganAMNIST (58,850 abdominal CT images, 11 classes), standard size-optimized RAPS converges to near-deterministic behavior with stratified undercoverage on uncertain samples, while our method achieves 95.72 percent global coverage with average set size 1.09 and at least 90 percent coverage across all strata. Cross-domain validation on PathMNIST (107,180 pathology images, 9 classes) confirms generalizability.
Quantitative Grad-CAM analysis (rho = -0.30, p < 1e-22) shows that multi-label predictions correspond to focused attention on anatomically ambiguous regions. These results demonstrate that the proposed method improves reliability while maintaining efficiency, making it suitable for safety-critical medical AI applications.
MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification
Department of Diagnostic and Interventional Radiology of the University Hospital RWTH Aachen（鲁尔大学亚琛医院诊断与介入放射科）；Else Kroener Fresenius Center for Digital Health, Technical University Dresden（德累斯顿技术大学埃勒克罗纳弗雷斯内尔数字健康中心）；Department of Medicine III, University Hospital RWTH Aachen（鲁尔大学亚琛医院第三医学部）；Pathology & Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds（利兹大学圣詹姆斯医院医学研究所以及病理学与数据科学）；Department of Medicine I, University Hospital Dresden（德累斯顿大学第一医学部）；Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg（海德堡大学国家肿瘤疾病中心（NCT）肿瘤医学部）
Deep learning methods have been very effective for a variety of medical diagnostic tasks and has even beaten human experts on some of those. However, the black-box nature of the algorithms has restricted clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clini…
A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳香槟校区联合学院）；Zhejiang University（浙江大学）；National University of Singapore（新加坡国立大学）；Angelalign Inc.（时代天使公司）
Generalizable and Explainable Deep Learning for Medical Image Computing: An Overview
School of Artificial Intelligence, Guilin University of Electronic Technology（桂林电子科技大学人工智能学院）；École de Technologie Supérieure (ETS)（高等技术学院）；College of Computer Science and Engineering, Taibah University（塔伊布大学计算机科学与工程学院）；College of Computer Science and Engineering, Jeddah University（吉达大学计算机科学与工程学院）
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所IIS）；Nuremberg Institute of Technology Georg Simon Ohm（纽伦堡乔治·西蒙·欧姆技术学院）
Would you trust physicians if they cannot explain their decisions to you? Medical diagnostics using machine learning gained enormously in importance within the last decade. However, without further enhancements many state-of-the-art machine learning methods are not suitable for medical application. The most important reasons are insufficient data set quality and the black-box behavior of machine learning algorithms such as Deep Learning models. Consequently, end-users cannot correct the model's decisions and the corresponding explanations. The latter is crucial for the trustworthiness of machi…
Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心影像科学研究所）；Utrecht University（乌得勒支大学）
With an increase in deep learning-based methods, the call for explainability of such methods grows, especially in high-stakes decision making areas such as medical image analysis. This survey presents an overview of eXplainable Artificial Intelligence (XAI) used in deep learning-based medical image analysis. A framework of XAI criteria is introduced to classify deep learning-based medical image analysis methods. Papers on XAI techniques in medical image analysis are then surveyed and categorized according to the framework and according to anatomical location. The paper concludes with an outloo…
Aalto University（阿尔托大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）
In the present paper we present the potential of Explainable Artificial Intelligence methods for decision-support in medical image analysis scenarios. With three types of explainable methods applied to the same medical image data set our aim was to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). The visual explanations were provided on in-vivo gastral images obtained from a Video capsule endoscopy (VCE), with the goal of increasing the health professionals' trust in the black box predictions. We implemented two post-hoc interpretable machine l…
Soft-CAM: Making black box models self-explainable for medical image analysis
Hertie Institute for AI in Brain Health, University of Tübingen, Germany（人工智能与脑健康赫尔特研究所，图宾根大学，德国）；Tübingen AI Center, University of Tübingen, Germany（图宾根人工智能中心，图宾根大学，德国）
Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes t…
Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks
Hong Kong University of Science and Technology（香港科技大学）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Harvard University（哈佛大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Indian Institute of Technology (IIT) Ropar（印度理工学院罗巴尔分校）；Monash University（莫纳什大学）；University of Canberra（堪培拉大学）；Inception Institute of Artificial Intelligence（因塞普申人工智能研究院）
We present a novel framework for explainable labeling and interpretation of medical images. Medical images require specialized professionals for interpretation, and are explained (typically) via elaborate textual reports. Different from prior methods that focus on medical report generation from images or vice-versa, we novelly generate congruent image--report pairs employing a cyclic-Generative Adversarial Network (cycleGAN); thereby, the generated report will adequately explain a medical image, while a report-generated image that effectively characterizes the text visually should (sufficientl…
Simon Fraser University（西蒙弗雷泽大学）；The University of British Columbia（不列颠哥伦比亚大学）
Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1)…
Medical image quality assessment is an important aspect of image acquisition, as poor-quality images may lead to misdiagnosis. Manual labelling of image quality is a tedious task for population studies and can lead to misleading results. While much research has been done on automated analysis of image quality to address this issue, relatively little work has been done to explain the methodologies. In this work, we propose an explainable image quality assessment system and validate our idea on two different objectives which are foreign object detection on Chest X-Rays (Object-CXR) and Left Vent…
SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging
Florida Institute of Technology（佛罗里达理工学院）；Knovel Engineering Lab（诺维尔工程实验室）；Vietnam Military Medical University（越南军事医科大学）；Military Central Hospital（108陆军中央医院）；Can Tho University of Medicine and Pharmacy（芹苴医药大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Training-free Test-time Improvement for Explainable Medical Image Classification
Department of Biomedical Engineering, Peking University, Beijing, China（北京大学生物医学工程系）；Institute of Medical Technology, Peking University, Beijing, China（北京大学医学技术研究所）；National Biomedical Imaging Center, Peking University, Beijing, China（北京大学国家生物医学成像中心）；Beijing Institute of Technology, Beijing, China（北京理工大学）；Equal Contribution（同等贡献）；Corresponding Author（通讯作者）
University of Central Florida（中佛罗里达大学）；University of Pennsylvania（宾夕法尼亚大学）
Convolutional neural network based systems have largely failed to be adopted in many high-risk application areas, including healthcare, military, security, transportation, finance, and legal, due to their highly uninterpretable "black-box" nature. Towards solving this deficiency, we teach a novel multi-task capsule network to improve the explainability of predictions by embodying the same high-level language used by human-experts. Our explainable capsule network, X-Caps, encodes high-level visual object attributes within the vectors of its capsules, then forms predictions based solely on these…
Transparency in Machine Learning (ML), attempts to reveal the working mechanisms of complex models. Transparent ML promises to advance human factors engineering goals of human-centered AI in the target users. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e. a relationship between algorithm and user; as a result, iterative prototyping and evaluation with users is critical to attaining adequate solutions that afford transparency. However, following human-centered design principles in healthcare and medical image analysis is challeng…
Bridging visual saliency and large language models for explainable deep learning in medical imaging
Department of Mathematics and Computer science（数学与计算机科学系）；Institute of Cognitive Science（认知科学研究所）；Rhodes University（罗德斯大学）；National Institute for Theoretical and Computational Sciences (NITheCS)（理论与计算科学国家研究所）
Athinoula A. Martinos Center for Biomedical Imaging（阿西娜·A·马蒂诺斯生物医学影像中心）；Massachusetts Institute of Technology（麻省理工学院）；Harvard Medical School（哈佛医学院）
Deep learning has the potential to automate many clinically useful tasks in medical imaging. However translation of deep learning into clinical practice has been hindered by issues such as lack of the transparency and interpretability in these "black box" algorithms compared to traditional statistical methods. Specifically, many clinical deep learning models lack rigorous and robust techniques for conveying certainty (or lack thereof) in their predictions -- ultimately limiting their appeal for extensive use in medical decision-making. Furthermore, numerous demonstrations of algorithmic bias h…
Deep learning has now become the de facto approach to the recognition of anomalies in medical imaging. Their 'black box' way of classifying medical images into anomaly labels poses problems for their acceptance, particularly with clinicians. Current explainable AI methods offer justifications through visualizations such as heat maps but cannot guarantee that the network is focusing on the relevant image region fully containing the anomaly. In this paper, we develop an approach to explainable AI in which the anomaly is assured to be overlapping the expected location when present. This is made p…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）；University Hospitals Leuven（鲁汶大学医院）；King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
Accurate medical image segmentation is essential for diagnosis and treatment planning of diseases. Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they are still challenged by complicated conditions where the segmentation target has large variations of position, shape and scale, and existing CNNs have a poor explainability that limits their application to clinical decisions. In this work, we make extensive use of multiple attentions in a CNN architecture and propose a comprehensive attention-based CNN (CA-Net) f…
Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation
City University of Hong Kong（香港城市大学）；Shenzhen University（深圳大学）；University of Rochester（罗切斯特大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；The Chinese University of Hong Kong（香港中文大学）
BRAC University（BRAC大学）；Islamic University of Technology（伊斯兰技术大学）
In the world of medical diagnostics, the adoption of various deep learning techniques is quite common as well as effective, and its statement is equally true when it comes to implementing it into the retina Optical Coherence Tomography (OCT) sector, but (i)These techniques have the black box characteristics that prevent the medical professionals to completely trust the results generated from them (ii)Lack of precision of these methods restricts their implementation in clinical and complex cases (iii)The existing works and models on the OCT classification are substantially large and complicated…
Despite the recent successes of deep learning, such models are still far from some human abilities like learning from few examples, reasoning and explaining decisions. In this paper, we focus on organ annotation in medical images and we introduce a reasoning framework that is based on learning fuzzy relations on a small dataset for generating explanations. Given a catalogue of relations, it efficiently induces the most relevant relations and combines them for building constraints in order to both solve the organ annotation task and generate explanations. We test our approach on a publicly avai…
Understanding model predictions is critical in healthcare, to facilitate rapid verification of model correctness and to guard against use of models that exploit confounding variables. We introduce the challenging new task of explainable multiple abnormality classification in volumetric medical images, in which a model must indicate the regions used to predict each abnormality. To solve this task, we propose a multiple instance learning convolutional neural network, AxialNet, that allows identification of top slices for each abnormality. Next we incorporate HiResCAM, an attention mechanism, to …
TWLR: Text-Guided Weakly-Supervised Lesion Localization and Severity Regression for Explainable Diabetic Retinopathy Grading
Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science（广东省级交叉学科研究与数据科学应用重点实验室）；Department of Statistics and Data Science, Beijing Normal-Hong Kong Baptist University（北京师范大学-香港 Baptist大学统计与数据科学系）；Faculty of Science, Hong Kong Baptist University（香港 Baptist大学科学学院）；Data Science Research Center, Duke Kunshan University（杜克-昆山大学数据科学研究中心）；Shanxi Provincial People’s Hospital（山西人民医院）；The Fifth Clinical Medical school of Shanxi Medical University（山西医科大学第五临床医学院）；Research Center for Mathematics, Beijing Normal University（北京师范大学数学研究中心）；Department of Mathematics and Statistics, York University（约克大学数学与统计学系）
Center for Public Health Genomics（公共卫生基因组学中心）；Department of Biomedical Engineering University of Virginia（弗吉尼亚大学生物医学工程系）；University of Virginia（弗吉尼亚大学）
White Blood Cell (WBC) Leukaemia is detected through image-based classification. Convolutional Neural Networks are used to learn the features needed to classify images of cells a malignant or healthy. However, this type of model requires learning a large number of parameters and is difficult to interpret and explain. Explainable AI (XAI) attempts to alleviate this issue by providing insights to how models make decisions. Therefore, we present an XAI model which uses only 24 explainable and interpretable features and is highly competitive to other approaches by outperforming them by about 4.38\…
MVP-CBM:Multi-layer Visual Preference-enhanced Concept Bottleneck Model for Explainable Medical Image Classification
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（中国科学技术大学生物医学工程学院、生命科学与医学系）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏等候研院）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术省级重点实验室）；Medical Business Department, iFlytek Co., Ltd（iFlytek有限公司医疗业务部）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（精准与智能化学国家重点实验室）
Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysis
School of Electrical and Computer Engineering, Cornell University（电气与计算机工程学院，康奈尔大学）；School of Computation, Information and Technology, Technical University of Munich（计算、信息与技术学院，慕尼黑技术大学）；Cornell Tech（康奈尔科技）；Weill Cornell Medicine（韦尔·康奈尔医学）
Learning to Select Like Humans: Explainable Active Learning for Medical Imaging
AI Research, Department of Computer Science, University of South Dakota（人工智能研究，计算机科学系，南达科他大学）；Department of Computer Science and Software Engineering, Auburn University（计算机科学与软件工程系，阿伯丁大学）
