OMGM：编排多种粒度与模态以实现高效的多模态检索
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
AI总结：
本文提出一种从粗到细多步检索的多模态RAG系统OMGM，通过协调多种粒度与模态，结合跨模态检索、多模态融合重排序与文本重排序，在InfoSeek和Encyclopedic-VQA基准上实现了最先进的检索性能与极具竞争力的回答结果。
摘要：
视觉语言检索增强生成（RAG）已成为解决基于知识的视觉问答（KB-VQA）的有效方法，该任务需要图像呈现的视觉内容之外的外部知识。视觉语言RAG系统的有效性取决于多模态检索，由于查询和知识库中存在多样的模态和知识粒度，这本身就具有挑战性。现有方法尚未充分利用这些元素之间的潜在交互作用。我们提出了一种多模态RAG系统，具有从粗到细的多步检索功能，协调多种粒度和模态以提高效能。我们的系统从广泛的初始搜索开始，对齐知识粒度以进行跨模态检索，随后进行多模态融合重排序，以捕获细致的多模态信息来选择顶级实体。然后，文本重排序器过滤出最相关的细粒度部分用于增强生成。在InfoSeek和Encyclopedic-VQA基准上的大量实验表明，我们的方法实现了最先进的检索性能和极具竞争力的回答结果，凸显了其在推进KB-VQA系统方面的有效性。
英文摘要：
Vision-language retrieval-augmented generation (RAG) has become an effective approach for tackling Knowledge-Based Visual Question Answering (KB-VQA), which requires external knowledge beyond the visual content presented in images. The effectiveness of Vision-language RAG systems hinges on multimodal retrieval, which is inherently challenging due to the diverse modalities and knowledge granularities in both queries and knowledge bases. Existing methods have not fully tapped into the potential interplay between these elements. We propose a multimodal RAG system featuring a coarse-to-fine, multi-step retrieval that harmonizes multiple granularities and modalities to enhance efficacy. Our system begins with a broad initial search aligning knowledge granularity for cross-modal retrieval, followed by a multimodal fusion reranking to capture the nuanced multimodal information for top entity selection. A text reranker then filters out the most relevant fine-grained section for augmented generation. Extensive experiments on the InfoSeek and Encyclopedic-VQA benchmarks show our method achieves state-of-the-art retrieval performance and highly competitive answering results, underscoring its effectiveness in advancing KB-VQA systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
