MOTOR：医学视觉问答中基于接地检索的多模态最优传输
MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answering
AI总结：
针对医学视觉问答中检索上下文不相关的问题，本文提出MOTOR方法，利用接地字幕和最优传输进行多模态检索与重排序，在MedVQA数据集上平均超越现有最优方法6.45%。
摘要：
医学视觉问答（MedVQA）通过为基于图像的查询提供上下文丰富的答案，在临床决策中发挥着重要作用。尽管视觉语言模型（VLM）被广泛用于该任务，但它们经常生成事实不正确的答案。检索增强生成通过提供来自外部来源的信息来应对这一挑战，但存在检索到不相关上下文的风险，这可能降低VLM的推理能力。现有方法中引入的重排序检索通过关注查询与文本的对齐来增强检索相关性。然而，这些方法忽略了视觉或多模态上下文，而这对医学诊断尤为关键。我们提出MOTOR，一种新颖的多模态检索与重排序方法，利用接地字幕（grounded captions）和最优传输。它基于文本和视觉信息捕捉查询与检索到的上下文之间的潜在关系。因此，我们的方法能够识别出更具临床相关性的上下文，以增强VLM的输入。实证分析和人类专家评估表明，MOTOR在MedVQA数据集上取得了更高的准确率，平均比最先进方法高出6.45%。代码可在https://github.com/BioMedIA-MBZUAI/MOTOR获取。
英文摘要：
Medical visual question answering (MedVQA) plays a vital role in clinical decision-making by providing contextually rich answers to image-based queries. Although vision-language models (VLMs) are widely used for this task, they often generate factually incorrect answers. Retrieval-augmented generation addresses this challenge by providing information from external sources, but risks retrieving irrelevant context, which can degrade the reasoning capabilities of VLMs. Re-ranking retrievals, as introduced in existing approaches, enhances retrieval relevance by focusing on query-text alignment. However, these approaches neglect the visual or multimodal context, which is particularly crucial for medical diagnosis. We propose MOTOR, a novel multimodal retrieval and re-ranking approach that leverages grounded captions and optimal transport. It captures the underlying relationships between the query and the retrieved context based on textual and visual information. Consequently, our approach identifies more clinically relevant contexts to augment the VLM input. Empirical analysis and human expert evaluation demonstrate that MOTOR achieves higher accuracy on MedVQA datasets, outperforming state-of-the-art methods by an average of 6.45%. Code is available at https://github.com/BioMedIA-MBZUAI/MOTOR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
