多模态重排序用于知识密集型视觉问答
Multimodal Reranking for Knowledge-Intensive Visual Question Answering
AI总结：
本文提出多模态重排序器，通过跨项交互改善知识候选排序，提升知识密集型视觉问答性能，实验验证其有效性。
摘要：
知识密集型视觉问答要求模型有效利用外部知识来帮助回答视觉问题。典型的流程包括知识检索器和答案生成器。然而，利用局部信息（如图像块）的检索器可能无法提供可靠的问答候选相关性分数。此外，双塔架构也限制了检索器对相关性分数建模的能力，难以选出最佳候选供答案生成器推理。在本文中，我们引入了一个额外的模块——多模态重排序器，以提高知识候选的排序质量，从而促进答案生成。我们的重排序模块从候选和问题中获取多模态信息，并进行跨项交互，以更好地建模相关性分数。在OK-VQA和A-OKVQA上的实验表明，通过远程监督训练的多模态重排序器能够带来一致的性能提升。我们还发现，在答案生成中，重排序存在训练与测试的不一致，当训练时的知识候选与测试时使用的相似或更嘈杂时，性能会有所提升。
英文摘要：
Knowledge-intensive visual question answering requires models to effectively use external knowledge to help answer visual questions. A typical pipeline includes a knowledge retriever and an answer generator. However, a retriever that utilizes local information, such as an image patch, may not provide reliable question-candidate relevance scores. Besides, the two-tower architecture also limits the relevance score modeling of a retriever to select top candidates for answer generator reasoning. In this paper, we introduce an additional module, a multi-modal reranker, to improve the ranking quality of knowledge candidates for answer generation. Our reranking module takes multi-modal information from both candidates and questions and performs cross-item interaction for better relevance score modeling. Experiments on OK-VQA and A-OKVQA show that multi-modal reranker from distant supervision provides consistent improvements. We also find a training-testing discrepancy with reranking in answer generation, where performance improves if training knowledge candidates are similar to or noisier than those used in testing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
