arXivDaily arXiv每日学术速递 周一至周五更新
​通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2604.27720cs.AI

对前沿视觉-语言模型进行审计以实现可信的医学视觉问答：定位失败、格式崩溃和领域适应

Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

New York University, New York, USA（纽约大学）；Tsinghua University, Beijing, China（清华大学）；Columbia University, New York, USA（哥伦比亚大学）；University of Michigan, Ann Arbor, USA（密歇根大学）

Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文审计了五种前沿视觉-语言模型在医学视觉问答中的表现，发现定位失败和格式崩溃是信任瓶颈，通过领域适应可提升性能。

摘要：

在临床环境中部署视觉-语言模型（VLMs）需要在现实故障条件下具有可审计行为，但前沿VLMs在专门医疗输入上的故障景观尚不明确。我们审计了五种近期前沿且具有定位意识的VLMs（Gemini~2.5~Pro、GPT-5、o3、GLM-4.5V、Qwen~2.5~VL）在医学VQA上的表现，从两个与信任相关轴线进行评估。感知：所有模型对解剖学和病理学目标的定位均不佳——最佳模型仅达到0.23均值IoU和19.1% Acc@0.5，并表现出临床危险的方位混淆。流程整合：一种自我定位流程，其中同一模型先定位后回答，会降低VQA准确性，驱动因素是不准确的定位和在两步提示下的格式合规性失败（在VQA-RAD上解析失败上升至70%至99% for Gemini和GPT-5）。用地面真实注释替换预测框可恢复并提高VQA准确性，与故障位于感知模块而非分解本身一致。这些观察性发现将定位质量确定为我们在SLAKE边界框设置中的主要信任瓶颈。作为补充微调后续，对Qwen~2.5~VL进行监督微调，使用结合的Med-VQA训练数据，达到最高报告的SLAKE开放式回忆（85.5%） among comparable methods，表明VQA层面的差距可通过领域适应解决；是否这也关闭了感知/信任瓶颈还需未来工作。

英文摘要：

Vision-language models (VLMs) are increasingly applied to medical visual question answering (Med-VQA), yet whether they can \emph{localize} the evidence behind their answers---a prerequisite for clinical auditability---is poorly characterized. We separately evaluate VQA reasoning and visual grounding for four recent frontier VLMs (GPT-5.1, GPT-5.5, Gemini-2.5-Pro, Gemini-3-Flash), two domain-specific medical VLMs (Lingshu, MedGemma), and a dedicated open-vocabulary detector (Grounding DINO) on VQA-RAD and SLAKE. Two findings challenge the intuition that ``add grounding to improve VQA.'' First, \textbf{no model localizes medical targets well}: every off-the-shelf system---frontier, medical-specialized, or dedicated detector---scores mean IoU 0.05--0.24 on our SLAKE grounding split, at or barely above a trivial center-box baseline (0.10), with Acc@0.5 below 20\%. Second, and counter to the common ``localize-then-answer'' paradigm, \textbf{cropping to a bounding box degrades VQA even when the box is a perfect oracle}: on the matched subset where oracle ground-truth boxes are applied, GT-grounding \emph{lowers} closed-ended accuracy for every model (by 0.9--18.0 points versus using the full image)---consistent with the crop discarding global context the model relies on. Because the oracle box removes localization error by construction, the problem is not that perception is a recoverable bottleneck, but that grounding-by-cropping is itself the wrong interface. Finally, we show constructively that the two channels need not conflict: supervised fine-tuning of Qwen-2.5-VL-7B on answers \emph{alone} silently destroys box-evidence emission (0/418 parseable boxes), whereas mixing in a small amount of grounding supervision restores localization to 0.36 IoU---above every zero-shot model---while preserving answer accuracy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑