arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2511.19220cs.CVcs.AI

大视觉语言模型真的在医学图像上具有基础性吗?来自意大利临床视觉问答的证据

Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering

SIIAM;NSBProject;Dept. of Life Sciences & Public Health, UCSC(生命科学与公共卫生系,UCSC);ASL RM 4;UCSC;Univ. Paris Cité(巴黎Cité大学);Univ. of Pisa(比萨大学)
Federico Felizzi, Olivia Riccomi, Michele Ferramola, Francesco Andrea Causio, Manuel Del Medico, Vittorio De Vita, Lorenzo De Mori, Alessandra Piscitelli, Pietr… 展开作者

Federico Felizzi, Olivia Riccomi, Michele Ferramola, Francesco Andrea Causio, Manuel Del Medico, Vittorio De Vita, Lorenzo De Mori, Alessandra Piscitelli, Pietro Eric Risuleo, Bianca Destro Castaniti, Antonio Cristiano, Alessia Longo, Luigi De Angelis, Mariapia Vassalli, Marcello Di Pumpo

AI总结:

研究通过测试四种先进模型在意大利医学问题上的表现,揭示了大视觉语言模型在视觉基础上的差异,强调了临床部署前的严格评估需求。

摘要:

大视觉语言模型(VLMs)在医学视觉问答基准测试中取得了令人印象深刻的表现,但其对视觉信息的依赖仍然不明确。我们调查了前沿VLMs在回答意大利医学问题时是否真正具有视觉基础,通过测试四种最先进的模型:Claude Sonnet 4.5、GPT-4o、GPT-5-mini和Gemini 2.0 flash exp。使用来自EuropeMedQA意大利数据集的60个问题,这些问题明确要求图像解释,我们用空白占位符替换正确的医学图像,以测试模型是否真正整合了视觉和文本信息。我们的结果揭示了视觉依赖性的显著变化:GPT-4o在视觉基础方面最强,准确率下降27.9pp(83.2% [74.6%, 91.7%]到55.3% [44.1%, 66.6%]),而GPT-5-mini、Gemini和Claude保持高准确率,分别下降8.5pp、2.4pp和5.6pp。对模型生成的推理分析显示,所有模型都对伪造的视觉解释提供了自信的解释,表明在依赖文本捷径与真正的视觉分析方面存在不同程度的依赖。这些发现突显了模型鲁棒性方面的关键差异,并强调了在临床部署前需要严格评估的必要性。

英文摘要:

Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑