自动化报告驱动的肿瘤学VQA基准：用于评估3D医学影像上的视觉-语言模型
Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
AI总结：
提出一个自动化管道，从私有放射学报告和3D肿瘤影像生成多选VQA数据集，构建无污染基准，评估六种视觉-语言模型，发现视觉依赖因数据集而异。
摘要：
评估医学影像上的视觉-语言模型（VLM）需要临床基础、可扩展且控制评估混淆的基准。现有的公共基准在规模上有限、需要手动标注，或可能泄露到VLM预训练语料中。我们提出一个自动化智能体驱动的管道，直接从配对的私有放射学报告和3D肿瘤影像生成多选VQA数据集，产生两种互补的问题类型：从临床医生定义的报告模式确定性导出的RADS风格问题，以及由LLM根据放射科医生发现生成并对照源报告验证的放射学报告衍生问题。应用于四个内部癌症队列，该管道产生一个实例污染控制的基准，无需每个问题的人工标注。对六个VLM的零样本评估显示没有主导模型，且所有单元均有显著提升空间。一项盲消融实验显示，视觉依赖高度特定于数据集：肝脏报告衍生问题确实需要图像，而肺CT基本上可以在没有图像的情况下解决——领先的闭源模型在盲测时在肺CT上的准确率超过其有视觉的准确率，这表明即使是私有临床数据也不能保证对视觉能力的污染控制读取。该管道作为开放智能体技能发布，用于内部重新部署。
英文摘要：
Evaluating vision-language models (VLMs) on medical images requires benchmarks that are clinically grounded, scalable, and controlled for evaluation confounds. Existing public benchmarks are limited in scale, manually annotated, or potentially leaked into VLM pretraining corpora. We present an automated agent-driven pipeline that generates multiple-choice VQA datasets directly from paired private radiology reports and 3D oncology imaging, producing two complementary question types: RADS-style questions deterministically derived from clinician-defined reporting schemas, and radiology report-derived questions generated by an LLM from radiologist findings and verified against the source report. Applied to four in-house cancer cohorts, the pipeline yields an instance-contamination-controlled benchmark without per-question human annotation. Zero-shot evaluation of six VLMs reveals no dominant model and substantial headroom across all cells. A blind ablation reveals that visual reliance is highly dataset-specific: liver Report-derived questions genuinely require the image, while Lung CT is essentially solvable without it - the leading closed model exceeds its sighted accuracy on Lung CT when blinded - indicating that even private clinical data does not guarantee a contamination-controlled read of visual capability. The pipeline is released as an open agent skill for in-house redeployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
