arXiv 2505.189152025-05-25cs.CV

视觉语言模型是否已准备好用于临床诊断？面向肿瘤中心视觉问答的3D医学基准

Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering

Johns Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia（意大利理工学院生物分子纳米技术中心）

Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi, Xinze Zhou, Sezgin Er, Ibrahim Ethem Hamamci, Zongwei Zhou, Alan Yuille

备注：: NeurIPS 2025 datasets&benchmarks track submission

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出DeepTumorVQA，一个含9,262个CT体、395K专家级问题的3D医学VQA基准，评估四个VLM，发现其测量尚可但识别与推理不足，且预训练和视觉模块设计是关键。

摘要：

视觉语言模型（VLMs）在多种2D视觉任务中展现出潜力，但由于对识别精度、推理能力和领域知识的严格要求，其是否准备好用于3D临床诊断仍不明确。为系统评估这些维度，我们提出了DeepTumorVQA，一个针对CT扫描中腹部肿瘤的诊断性视觉问答（VQA）基准。该基准包含来自17个公共数据集的9,262个CT体（370万张切片），以及涵盖四个类别的395,000个专家级问题：识别、测量、视觉推理和医学推理。DeepTumorVQA引入了独特挑战，包括小肿瘤检测和跨3D解剖结构的临床推理。通过对四个先进VLM（RadFM、M3D、Merlin、CT-CHAT）进行基准测试，我们发现当前模型在测量任务上表现尚可，但在病变识别和推理方面存在困难，且仍不能满足临床需求。两个关键见解浮现：（1）大规模多模态预训练在DeepTumorVQA测试性能中起关键作用，使RadFM在所有VLM中脱颖而出。（2）我们的数据集揭示了VLM组件中的关键差异，其中适当的图像预处理和视觉模块设计显著影响3D感知。为促进医学多模态研究，我们已发布DeepTumorVQA作为严格基准：https://github.com/Schuture/DeepTumorVQA。

英文摘要：

Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。