arXiv 2509.211022025-09-25cs.CV

Mammo-CLIP Dissect：用于分析视觉语言模型中乳腺摄影概念的一个框架

Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models

UiT The Arctic University of Norway（乌塔大学极地大学）；Technical University of Denmark（技术大学）；Østfold Hospital Trust（奥斯fold医院信托）；Vestre Viken Hospital Trust（维斯特维肯医院信托）；Radboud University Nijmegen Medical Centre（拉德堡德大学奈梅亨医疗中心）；The Netherlands Cancer Institute（荷兰癌症研究所）；Antoni van Leeuwenhoek Hospital（安东尼·弗莱明医院）；University of Copenhagen（哥本哈根大学）

Suaiba Amina Salahuddin, Teresa Dorszewski, Marit Almenning Martiniussen, Tone Hovda, Antonio Portaluri, Solveig Thrun, Michael Kampffmeyer, Elisabeth Wetzer, K… 展开作者

Suaiba Amina Salahuddin, Teresa Dorszewski, Marit Almenning Martiniussen, Tone Hovda, Antonio Portaluri, Solveig Thrun, Michael Kampffmeyer, Elisabeth Wetzer, Kristoffer Wickstrøm, Robert Jenssen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Mammo-CLIP Dissect框架，用于剖析乳腺摄影视觉语言模型中的概念学习，发现领域特定训练提升临床相关性，但微调带来特化与泛化的权衡。

摘要：

理解深度学习（DL）模型所学到的内容对于人工智能（AI）在临床环境中的安全部署至关重要。虽然以往的工作侧重于基于像素的可解释性方法，但较少关注这些模型学到的文本概念，而这些概念可能更好地反映临床医生使用的推理方式。我们引入了Mammo-CLIP Dissect，这是第一个基于概念的可解释性框架，用于系统地剖析针对乳腺摄影训练的DL视觉模型。利用一个专门针对乳腺摄影的视觉语言模型（Mammo-CLIP）作为“解剖器”，我们的方法用人类可解释的文本概念标记指定层中的神经元，并量化它们与领域知识的一致性。使用Mammo-CLIP Dissect，我们研究了三个关键问题：（1）在通用图像数据集上训练的DL视觉模型与在乳腺摄影特定数据集上训练的模型之间，概念学习有何不同；（2）针对下游乳腺摄影任务的微调如何影响概念特化；（3）哪些与乳腺摄影相关的概念仍然代表性不足。我们表明，在乳腺摄影数据上训练的模型比未在乳腺摄影数据上训练的模型捕获更多临床相关的概念，并与放射科医生的工作流程更紧密地对齐。针对任务特定分类的微调增强了某些概念类别（例如，良性钙化）的捕获，但可能减少其他概念（例如，密度相关特征）的覆盖，表明特化与泛化之间存在权衡。我们的发现表明，Mammo-CLIP Dissect提供了关于卷积神经网络（CNN）如何捕获乳腺摄影特定知识的见解。通过比较不同训练数据和微调机制下的模型，我们揭示了领域特定训练和任务特定适应如何塑造概念学习。代码和概念集可在以下网址获取：https://github.com/Suaiba/Mammo-CLIP-Dissect。

英文摘要：

Understanding what deep learning (DL) models learn is essential for the safe deployment of artificial intelligence (AI) in clinical settings. While previous work has focused on pixel-based explainability methods, less attention has been paid to the textual concepts learned by these models, which may better reflect the reasoning used by clinicians. We introduce Mammo-CLIP Dissect, the first concept-based explainability framework for systematically dissecting DL vision models trained for mammography. Leveraging a mammography-specific vision-language model (Mammo-CLIP) as a "dissector," our approach labels neurons at specified layers with human-interpretable textual concepts and quantifies their alignment to domain knowledge. Using Mammo-CLIP Dissect, we investigate three key questions: (1) how concept learning differs between DL vision models trained on general image datasets versus mammography-specific datasets; (2) how fine-tuning for downstream mammography tasks affects concept specialisation; and (3) which mammography-relevant concepts remain underrepresented. We show that models trained on mammography data capture more clinically relevant concepts and align more closely with radiologists' workflows than models not trained on mammography data. Fine-tuning for task-specific classification enhances the capture of certain concept categories (e.g., benign calcifications) but can reduce coverage of others (e.g., density-related features), indicating a trade-off between specialisation and generalisation. Our findings show that Mammo-CLIP Dissect provides insights into how convolutional neural networks (CNNs) capture mammography-specific knowledge. By comparing models across training data and fine-tuning regimes, we reveal how domain-specific training and task-specific adaptation shape concept learning. Code and concept set are available: https://github.com/Suaiba/Mammo-CLIP-Dissect.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。