arXiv 2309.171332023-10-28cs.CLcs.CV

面向检索增强视觉问答的细粒度晚期交互多模态检索

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

University of Cambridge（剑桥大学）

Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, Bill Byrne

备注：: To appear at NeurIPS 2023. This is the camera-ready version. We fixed some numbers and added more experiments to address reviewers' comments

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出细粒度晚期交互多模态检索FLMR，通过视觉模型对齐和多维嵌入改进RA-VQA的知识检索，将PRRecall@5提升约8%，并在OK-VQA上取得约61%的VQA分数。

摘要：

基于知识的视觉问答（KB-VQA）要求VQA系统利用外部知识库中的知识来回答以视觉为基础的问题。检索增强视觉问答（RA-VQA）是解决KB-VQA的一种强有力框架，它首先使用稠密段落检索（DPR）检索相关文档，然后利用这些文档回答问题。本文提出了细粒度晚期交互多模态检索（FLMR），显著改进了RA-VQA中的知识检索。FLMR解决了RA-VQA检索器的两个主要局限：（1）通过图像到文本转换获得的图像表示可能不完整且不准确；（2）查询与文档之间的相关性分数使用一维嵌入计算，可能对更细粒度的相关性不敏感。FLMR通过使用一个视觉模型来获取图像表示，该视觉模型通过一个简单的对齐网络与现有的基于文本的检索器对齐，从而补充图像到文本转换得到的表示，克服了这些局限。FLMR还使用多维嵌入对图像和问题进行编码，以捕获查询与文档之间更细粒度的相关性。FLMR将原始RA-VQA检索器的PRRecall@5显著提高了约8%。最后，我们为RA-VQA配备了两个最先进的大型多模态/语言模型，在OK-VQA数据集上取得了约61%的VQA分数。

英文摘要：

Knowledge-based Visual Question Answering (KB-VQA) requires VQA systems to utilize knowledge from external knowledge bases to answer visually-grounded questions. Retrieval-Augmented Visual Question Answering (RA-VQA), a strong framework to tackle KB-VQA, first retrieves related documents with Dense Passage Retrieval (DPR) and then uses them to answer questions. This paper proposes Fine-grained Late-interaction Multi-modal Retrieval (FLMR) which significantly improves knowledge retrieval in RA-VQA. FLMR addresses two major limitations in RA-VQA's retriever: (1) the image representations obtained via image-to-text transforms can be incomplete and inaccurate and (2) relevance scores between queries and documents are computed with one-dimensional embeddings, which can be insensitive to finer-grained relevance. FLMR overcomes these limitations by obtaining image representations that complement those from the image-to-text transforms using a vision model aligned with an existing text-based retriever through a simple alignment network. FLMR also encodes images and questions using multi-dimensional embeddings to capture finer-grained relevance between queries and documents. FLMR significantly improves the original RA-VQA retriever's PRRecall@5 by approximately 8\%. Finally, we equipped RA-VQA with two state-of-the-art large multi-modal/language models to achieve $\sim61\%$ VQA score in the OK-VQA dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。