WikiSeeker: 重新思考视觉语言模型在基于知识的视觉问答中的作用
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
AI总结：
本文提出WikiSeeker框架，通过引入多模态检索器和重新定义视觉语言模型的角色，提升多模态检索性能和答案质量，实现在EVQA、InfoSeek和M2KR数据集上的最优表现。
摘要：
多模态检索增强生成（RAG）已成为知识驱动的视觉问答（KB-VQA）中高度有效的范式。尽管近期取得了进展，现有方法仍主要依赖图像作为检索键，并往往忽视或误置视觉语言模型（VLMs）的作用，从而未能充分利用其潜力。本文引入WikiSeeker，一种新颖的多模态RAG框架，通过提出多模态检索器和重新定义VLMs的角色来弥合这些差距。与仅作为答案生成器不同，我们为VLMs分配了两个专门的代理：精炼器和检查器。精炼器利用VLMs的能力，根据输入图像重写文本查询，显著提高了多模态检索器的性能。检查器通过选择性地将可靠检索到的上下文路由到另一个LLM进行答案生成，同时在检索不可靠时依赖VLMs内部的知识。在EVQA、InfoSeek和M2KR上的大量实验表明，WikiSeeker实现了最先进的性能，检索准确性和答案质量均有显著提升。我们的代码将在https://github.com/zhuyjan/WikiSeeker上发布。
英文摘要：
Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
