SURf:教导大型视觉语言模型选择性利用检索信息
SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
AI总结:
针对大型视觉语言模型(LVLMs)难以选择性利用检索信息且易受无关或误导信息影响的问题,提出SURf自我精炼框架,通过结合正负参考资料微调模型,显著提升了其有效利用多模态检索信息的能力及鲁棒性。
摘要:
大型视觉语言模型(LVLMs)在计算机视觉与自然语言处理的交叉领域变得至关重要。然而,LVLMs检索增强生成(RAG)能力的全部潜力仍未被充分利用。现有工作要么仅关注文本模态,要么局限于特定任务。此外,大多数LVLMs难以选择性地利用检索信息,并且对无关或误导性的参考资料很敏感。为了应对这些挑战,我们提出了一个自我精炼框架,旨在教导LVLMs选择性利用检索信息(SURf)。具体而言,当给定LVLM主干回答错误的问题时,我们会获取有助于纠正答案的参考资料(正参考资料)以及无助于纠正答案的参考资料(负参考资料)。然后,我们使用这些正参考资料和负参考资料的组合来微调LVLM主干。我们在三个任务和七个数据集上的实验表明,我们的框架显著增强了LVLMs有效利用检索到的多模态参考资料的能力,并提高了它们对无关或误导信息的鲁棒性。源代码可在 https://github.com/GasolSun36/SURf 获取。
英文摘要:
Large Vision-Language Models (LVLMs) have become pivotal at the intersection of computer vision and natural language processing. However, the full potential of LVLMs Retrieval-Augmented Generation (RAG) capabilities remains underutilized. Existing works either focus solely on the text modality or are limited to specific tasks. Moreover, most LVLMs struggle to selectively utilize retrieved information and are sensitive to irrelevant or misleading references. To address these challenges, we propose a self-refinement framework designed to teach LVLMs to Selectively Utilize Retrieved Information (SURf). Specifically, when given questions that are incorrectly answered by the LVLM backbone, we obtain references that help correct the answers (positive references) and those that do not (negative references). We then fine-tune the LVLM backbone using a combination of these positive and negative references. Our experiments across three tasks and seven datasets demonstrate that our framework significantly enhances LVLMs ability to effectively utilize retrieved multimodal references and improves their robustness against irrelevant or misleading information. The source code is available at https://github.com/GasolSun36/SURf.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。