先定位后回答：一种用于基于地面的医学多模态大语言模型的幻觉评估基准
Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs
AI总结：
本文提出HEAL-MedVQA基准，通过创新评估协议和67K VQA数据集，评估医学多模态模型的定位能力与幻觉鲁棒性，提出LobA框架提升视觉推理能力，实验结果表明其在挑战性基准中优于现有生物医学LMMs。
摘要：
医学大型多模态模型（LMMs）在医学数据解释中表现出显著能力。然而，这些模型经常生成与源证据矛盾的幻觉，特别是由于定位推理不足。本工作揭示了当前医学LMMs的关键限制：在回答疾病相关问题时，它们不分析相关病理区域，而是依赖语言模式或关注无关图像区域。为解决此问题，我们引入HEAL-MedVQA（通过定位进行医学VQA的幻觉评估），一个全面的基准，用于评估LMMs的定位能力和幻觉鲁棒性。HEAL-MedVQA具有（i）两个创新的评估协议以评估视觉和文本捷径学习，以及（ii）包含67K VQA对的数据集，配有医生标注的解剖分割掩码以标记病理区域。为提高视觉推理能力，我们提出Localize-before-Answer（LobA）框架，训练LMMs定位目标区域并自我提示以强调分割的病理区域，生成基于事实的可靠回答。实验结果表明，我们的方法在挑战性的HEAL-MedVQA基准上显著优于最先进的生物医学LMMs，推动了医学VQA的鲁棒性。
英文摘要：
Medical Large Multi-modal Models (LMMs) have demonstrated remarkable capabilities in medical data interpretation. However, these models frequently generate hallucinations contradicting source evidence, particularly due to inadequate localization reasoning. This work reveals a critical limitation in current medical LMMs: instead of analyzing relevant pathological regions, they often rely on linguistic patterns or attend to irrelevant image areas when responding to disease-related queries. To address this, we introduce HEAL-MedVQA (Hallucination Evaluation via Localization MedVQA), a comprehensive benchmark designed to evaluate LMMs' localization abilities and hallucination robustness. HEAL-MedVQA features (i) two innovative evaluation protocols to assess visual and textual shortcut learning, and (ii) a dataset of 67K VQA pairs, with doctor-annotated anatomical segmentation masks for pathological regions. To improve visual reasoning, we propose the Localize-before-Answer (LobA) framework, which trains LMMs to localize target regions of interest and self-prompt to emphasize segmented pathological areas, generating grounded and reliable answers. Experimental results demonstrate that our approach significantly outperforms state-of-the-art biomedical LMMs on the challenging HEAL-MedVQA benchmark, advancing robustness in medical VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
