医学视觉-语言模型在放射学中学到了什么？分布偏移下的迁移、对齐与源代理泄露

中文摘要 AI 辅助

医学视觉-语言模型（VLMs）在域内表现看似可靠，但当采集域、配对监督或评估协议发生变化时会失效。我们将这种失效模式视为与认知智能相关的表征层面盲区，不声称对认知不确定性有正式估计量。使用NIH ChestXray14和CheXpert数据集，我们首先将仅源域的跨数据集视觉迁移与无监督域适应诊断隔离开来；再使用PadChest和OpenI数据集，在严格的配对索引检索下评估多模态对齐，并量化冻结嵌入中保留的源自元数据的源代理信息。在匹配的ResNet-18对比中，自监督视觉初始化比监督ImageNet初始化能提升NIH到CheXpert的迁移效果，而对抗适应仅在狭窄范围内有用，且随对抗压力增大变得不稳定。在外部OpenI压力测试下，多模态精确配对检索仍处于较低水平，且源代理信息仍可从学习到的表征中恢复。定性最近邻与Grad-CAM分析显示，多数情况下存在临床合理的跨数据集结构与胸部注意力模式，而设备密集型及假阳性案例仍不明确。辅助架构检查具有任务依赖性，不支持通用骨干网络排名。总体而言，本研究表明，单一协议下的表观能力可掩盖与迁移、对齐及捷径相关的失效模式，推动对医学VLMs在分布偏移下的压力测试评估。

英文摘要

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

发表机构

• University of Constantine（康斯坦丁大学）
• University of Dubai（迪拜大学）
• University of Western Brittany（西布列塔尼大学）

机构由 AI 辅助整理，请以论文原文为准。