arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.09679cs.CVcs.AI
收藏

DeepTumorVQA: 一种分层的3D CT基准,用于分阶段评估医学视觉语言模型和工具增强代理

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents

Johns Hopkins University(约翰霍普金斯大学);University of Bologna(博洛尼亚大学);Istanbul Medipol University(伊斯坦布尔梅迪波尔大学);Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia(生物分子纳米技术中心,意大利技术研究院);The First Affiliated Hospital, Sun Yat-Sen University(中山大学第一附属医院);Tongji University(同济大学)

Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi, Boyan Wang, Liang He, Xinze Zhou, Sezgin Er, Ibrahim Ethem Hamamci, Zongwei Zhou, Alan Yuille

AI总结:

DeepTumorVQA通过分阶段证据链分解3D CT推理为四个阶段,提供工具交互环境,揭示医疗VLMs的瓶颈并展示工具增强的改进。

摘要:

医疗视觉语言模型(VLMs)和AI代理在学习分析和推理临床图像方面取得了显著进展。然而,现有医学视觉问答(VQA)基准将模型能力折合成单一准确性分数,掩盖了模型失败的位置和原因。我们提出了DeepTumorVQA,一种分层基准,遵循肿瘤诊断的多阶段证据链,将3D CT推理分解为四个阶段:识别、测量、视觉推理和医学推理。高级问题仍可独立评分,而其真实证据链定义在低级原始元素上。该基准包含476K个问题,覆盖42种临床子类型,在9,262个3D CT体积上。除了为VLMs提供的直接推理模式外,DeepTumorVQA还提供工具交互环境用于代理评估,其中模型可在回答问题前调用外部工具,包括分割模型、测量程序和医学知识模块。评估30种模型配置后,发现可靠的定量测量是主要瓶颈,使VLMs的后期视觉和医学推理更困难,而工具增强显著缓解了这一问题。当工具可用时,利用医学知识和工具推理医疗图像成为新挑战。我们进一步展示,DeepTumorVQA中的真实逐步工具使用轨迹可监督代理并减少工具使用和推理失败。从识别到测量再到视觉和医学推理的阶段式进展为未来医疗VLM和AI代理研究提供了具体路线图。所有数据和代码均在https://github.com/Schuture/DeepTumorVQA发布。

英文摘要:

Medical vision-language models (VLMs) and AI agents have made significant progress in learning to analyze and reason about clinical images. However, existing medical visual question answering (VQA) benchmarks collapse model capabilities into a single accuracy score, obscuring where and why models fail. We propose DeepTumorVQA, a hierarchical benchmark that follows the multi-stage evidence chain in tumor diagnosis and decomposes 3D CT reasoning into four stages: recognition, measurement, visual reasoning, and medical reasoning. Higher-level questions remain independently scorable, while their ground-truth evidence chains are defined over lower-level primitives. The benchmark contains 476K questions across 42 clinical subtypes on 9,262 3D CT volumes. In addition to a direct reasoning mode for VLMs, DeepTumorVQA provides tool-interaction environments for agent evaluation, where a model can call external tools, including segmentation models, measurement programs, and medical knowledge modules, before answering the question. Evaluating over 30 model configurations, we find that reliable quantitative measurement is the primary bottleneck, making later-stage visual and medical reasoning harder for VLMs, while tool augmentation substantially mitigates this issue. When tools are available, leveraging medical knowledge and tools to reason about medical images becomes a new challenge. We further show that ground-truth step-by-step tool-use traces from DeepTumorVQA can supervise agents and reduce tool-use and reasoning failures. This stage-wise progression from recognition to measurement to visual and medical reasoning provides a concrete roadmap for future medical VLM and AI agent studies. All data and code are released at https://github.com/Schuture/DeepTumorVQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑