医学 MLLMs 如何失效？对医学图像中视觉语义关联的探讨
How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
AI总结：
本文研究了医学 MLLMs 在视觉关联上的不足，通过设计 VGMED 数据集和 VGRefine 方法，验证了其在医学图像任务中的性能瓶颈。
摘要：
通用多模态大语言模型（MLLMs）在广泛视觉-语言任务中表现出色，但在医疗任务，尤其是零样本设置中表现不佳。本文首次系统研究了最先进的医学 MLLMs 的视觉关联能力，设计了 VGMED 数据集，引入了新的定量指标并进行了定性分析。研究发现，这些模型常无法在临床相关图像区域进行有效关联。基于此，提出 VGRefine 方法，通过优化注意力分布提升医疗场景中的视觉关联能力，在六个多样化的 Med-VQA 评估中取得最佳性能，无需额外训练或外部专家模型。
英文摘要：
Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks, particularly in zero-shot settings where generalization is critical, remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle visual grounding from semantic grounding, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across eight state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images. Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance. Additional experiments are included in the Supp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
