弥合视觉显著性与大语言模型以实现医学影像中可解释的深度学习
Bridging visual saliency and large language models for explainable deep learning in medical imaging
AI总结：
本文提出一种多模态可解释框架，通过结合CNN和大语言模型，生成脑肿瘤分类的可操作临床见解，提升深度学习在医学影像中的透明度和临床责任性。
摘要：
深度学习模型的不透明性仍是其在医学影像中临床应用的主要障碍。本文提出一种多模态可解释框架，通过结合卷积神经网络（CNN）预测与临床可操作见解，利用大语言模型（LLMs）生成人类可理解的诊断叙述。该框架通过三个耦合阶段操作：首先，九种CNN架构扩展为双输出混合公式，同时优化分类头和分割头，实现更丰富的空间特征学习；其次，应用视觉显著性分配方法，如Grad-CAM、Grad-CAM++和ScoreCAM，生成类别判别性热图，并通过自适应百分位阈值管道精制为二进制肿瘤掩码；第三，将所得掩码映射到哈佛-牛津皮层图谱，将像素级证据转换为命名的神经解剖结构，并将提取的发现编码为结构化JSON文件，以条件三个LLM（Grok3、Mistral和LLaMA）生成连贯的放射科风格诊断报告。在包含4,834例对比增强T1加权脑部MRI图像的三类肿瘤数据集上评估，InceptionResNetV2实现了最高的分类性能，Grad-CAM++获得了最佳分割重叠。在语言模型中，Grok3在词汇多样性及连贯性上领先，而LLaMA获得了最高的可读性评分。通过将视觉、解剖学和语言学模态整合到统一的管道中，该框架生成了技术上可靠且有意义可解释的解释，推动了人工智能辅助脑肿瘤诊断的透明性和临床责任性。
英文摘要：
The opaque nature of deep learning models remains a significant barrier to their clinical adoption in medical imaging. This paper presents a multimodal explainability framework that bridges the gap between convolutional neural network (CNN) predictions and clinically actionable insights for brain tumor classification, leveraging large language models (LLMs) to deliver human-interpretable diagnostic narratives. The proposed framework operates through three coupled stages. First, nine CNN architectures are extended with a dual-output hybrid formulation that simultaneously optimises a classification head and a segmentation head, enabling spatially richer feature learning. Second, visual saliency attribution methods, namely Grad-CAM, Grad-CAM++, and ScoreCAM, are applied to generate class-discriminative heatmaps, which are subsequently refined into binary tumor masks via an adaptive percentile thresholding pipeline. Third, the resulting masks are mapped onto the Harvard-Oxford cortical atlas to translate pixel-level evidence into named neuroanatomical structures, and the extracted findings are encoded into a structured JSON file that conditions three LLMs (Grok3, Mistral, and LLaMA) to generate coherent, radiological-style diagnostic reports. Evaluated on a dataset of 4,834 contrast-enhanced T1-weighted brain MRI images spanning three tumor classes, InceptionResNetV2 achieved the highest classification performance and Grad-CAM++ yielded the best segmentation overlap. Among the language models, Grok3 led in lexical diversity and coherence, while LLaMA achieved the highest readability score. By integrating visual, anatomical, and linguistic modalities into a unified pipeline, the framework produces explanations that are technically grounded and meaningfully interpretable, advancing the transparency and clinical accountability of artificial intelligence assisted brain tumor diagnosis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
