发表机构
University of Tehran; Tehran University of Medical Sciences; Advanced Diagnostic and Interventional Radiology Research Center (ADIR)（德黑兰大学; 德黑兰医科大学; 高级诊断与介入放射学研究中心（ADIR））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过脑部MRI数据集评估四类视觉-语言模型在视觉、文本扰动下的诊断鲁棒性，发现其存在预测翻转、文本选择偏差、诊断过度承诺等问题，指出需采用稳定性指标评估其临床应用可靠性。
AI 中文摘要
视觉-语言模型（VLM）的标准准确率指标往往掩盖了其在敏感领域的严重可靠性缺陷。本研究采用经组织病理学验证的脑部MRI数据集，系统评估四类VLM在保留证据的扰动下的诊断鲁棒性。通过重排解剖切片顺序和交换目标标签位置，我们评估当临床证据保持不变时模型是否能维持一致的预测。结果显示，模型在呈现顺序稳定性上存在显著漏洞：在简单序列反转的情况下，多达48.9%的案例会出现预测翻转。我们还发现了文本选择偏差：尽管视觉输入完全相同，标签重排会导致多达67.8%的案例出现诊断不一致。阴性对照测试进一步揭示了诊断过度承诺：在移除专家标注的病变切片后，模型仍会在多达76.1%的案例中生成分类诊断。这些结果表明，高准确率可能高估临床可靠性，掩盖了对序列呈现和文本框架的敏感性，而这些敏感性并未被总体准确率所捕捉。我们的研究强调，在安全关键的临床应用中部署VLM时，必须采用基于稳定性的指标。我们的评估数据和代码将在论文接收后公开。
英文摘要
Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validated brain MRI dataset to systematically assess the diagnostic robustness of four VLM families under evidence-preserving perturbations. By reordering anatomical slices and swapping target label positions, we evaluate whether models maintain consistent predictions when clinical evidence remains invariant. Our results reveal significant vulnerabilities in presentation-order stability, with models exhibiting prediction flips in up to 48.9% of cases under simple sequence reversals. We further identify a textual selection bias, where label reordering triggers inconsistent diagnoses in up to 67.8% of cases despite identical visual inputs. Negative-control tests further reveal diagnostic overcommitment: models generate categorical diagnoses in up to 76.1% of cases after expert-annotated lesion slices are removed. These results demonstrate that high accuracy can overestimate clinical reliability, masking sensitivity to sequential presentation and textual framing that is not captured by aggregate accuracy. Our findings highlight the necessity of stability-based metrics for the deployment of VLMs in safety-critical clinical applications. Our evaluation data and code will be made public upon acceptance.
