通过视觉问答提示医学大型视觉-语言模型诊断病变
Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering
AI总结：
本文提出两种提示策略，通过病变解释和弱学习器判断缓解医学LVLM的幻觉与数据不平衡问题，显著提升VQA诊断性能并可推广至通用LVLM。
摘要：
近年来，大型视觉-语言模型（LVLMs）取得了显著成功，并已被扩展到医学领域。尽管医学LVLM（MLVLMs）在医学视觉问答（VQA）任务上表现出令人满意的性能，但它们受到幻觉问题的困扰，这使其无法诊断复杂病变。此外，由于训练数据不平衡，它们往往难以学习少数类病变。我们提出了两种用于MLVLM的提示策略，以减少幻觉并提高VQA性能。在第一种策略中，我们对所查询的病变提供详细解释。在第二种策略中，我们微调一个低成本的弱学习器，使其在特定指标上实现高性能，并以文本形式将其判断提供给MLVLM。在MIMIC-CXR-JPG和Chexpert数据集上测试时，我们的方法显著提高了诊断F1分数，最高提升达0.27。我们还证明，所提出的提示策略可以扩展到通用LVLM领域。基于POPE指标，该方法有效抑制了现有LVLM的假阴性预测，并将Recall提高约0.07。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved significant success in recent years, and they have been extended to the medical domain. Although demonstrating satisfactory performance on medical Visual Question Answering (VQA) tasks, Medical LVLMs (MLVLMs) suffer from the hallucination problem, which makes them fail to diagnose complex pathologies. Moreover, they readily fail to learn minority pathologies due to imbalanced training data. We propose two prompting strategies for MLVLMs that reduce hallucination and improve VQA performance. In the first strategy, we provide a detailed explanation of the queried pathology. In the second strategy, we fine-tune a cheap, weak learner to achieve high performance on a specific metric, and textually provide its judgment to the MLVLM. Tested on the MIMIC-CXR-JPG and Chexpert datasets, our methods significantly improve the diagnostic F1 score, with the highest increase being 0.27. We also demonstrate that our prompting strategies can be extended to general LVLM domains. Based on POPE metrics, it effectively suppresses the false negative predictions of existing LVLMs and improves Recall by approximately 0.07.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
