面向医学视觉问答的定向视觉提示
Targeted Visual Prompting for Medical Visual Question Answering
AI总结：
本文提出定向视觉提示方法，通过向多模态大语言模型同时展示孤立区域及其上下文，赋予其基于区域的医学视觉问答能力，并在多个数据集上验证了有效性。
摘要：
近年来，随着研究兴趣的不断增长，医学视觉问答（Med-VQA）迅速发展，多模态大语言模型（MLLMs）逐渐成为经典模型架构的替代方案。具体而言，这类模型能够将视觉信息添加到预训练大语言模型的输入中，从而为图像解读带来新的能力。然而，简单的视觉错误使人们对这些模型的实际视觉理解能力产生怀疑。为解决这一问题，基于区域的提问被提出，作为一种通过组合式评估来考察和增强实际视觉理解能力的手段。为了结合上述两种视角，本文引入定向视觉提示方法，使多模态大语言模型具备基于区域的提问能力。通过在定制化的视觉提示中同时向模型展示孤立区域及其所在上下文中的区域，我们在多个数据集上验证了该方法的有效性，并与多个基线模型进行了比较。我们的代码和数据可在 https://github.com/sergiotasconmorales/locvqallm 获取。
英文摘要：
With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models. Our code and data are available at https://github.com/sergiotasconmorales/locvqallm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
