arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.10037cs.CV
收藏

知识凝聚与推理用于基于知识的视觉问答

Knowledge Condensation and Reasoning for Knowledge-based VQA

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Kuaishou(快手)

Dongze Hao, Jian Jia, Longteng Guo, Qunbo Wang, Te Yang, Yan Li, Yanhua Cheng, Bo Wang, Quan Chen, Han Li, Jing Liu

AI总结:

针对基于知识的视觉问答中检索知识含噪问题,提出知识凝聚与推理协同模型,提炼简洁概念与知识精髓,实现最先进性能(OK-VQA 65.1%,A-OKVQA 60.1%)。

摘要:

基于知识的视觉问答(KB-VQA)是一项具有挑战性的任务,要求模型利用外部知识来理解并回答基于视觉内容的问题。近期研究从外部知识库中检索知识段落,然后使用这些段落来回答问题。然而,这些检索到的知识段落往往包含无关或嘈杂的信息,这限制了模型的性能。为了应对这一挑战,我们提出了两个协同模型:知识凝聚模型和知识推理模型。我们从两个角度凝聚检索到的知识段落。首先,我们利用视觉-语言模型的多模态感知和推理能力,从检索到的冗长段落中提炼出简洁的知识概念,确保这些概念与视觉内容和问题都相关。其次,我们利用大型语言模型的文本理解能力,将段落总结并凝聚为有助于回答问题的知识精髓。这两种类型的凝聚知识随后无缝集成到我们的知识推理模型中,该模型明智地导航于合并后的信息以得出最终答案。大量实验验证了所提出方法的优越性。与先前方法相比,我们的方法在基于知识的VQA数据集上实现了最先进的性能(在OK-VQA上达到65.1%,在A-OKVQA上达到60.1%),且无需借助GPT-3(175B)产生的知识。

英文摘要:

Knowledge-based visual question answering (KB-VQA) is a challenging task, which requires the model to leverage external knowledge for comprehending and answering questions grounded in visual content. Recent studies retrieve the knowledge passages from external knowledge bases and then use them to answer questions. However, these retrieved knowledge passages often contain irrelevant or noisy information, which limits the performance of the model. To address the challenge, we propose two synergistic models: Knowledge Condensation model and Knowledge Reasoning model. We condense the retrieved knowledge passages from two perspectives. First, we leverage the multimodal perception and reasoning ability of the visual-language models to distill concise knowledge concepts from retrieved lengthy passages, ensuring relevance to both the visual content and the question. Second, we leverage the text comprehension ability of the large language models to summarize and condense the passages into the knowledge essence which helps answer the question. These two types of condensed knowledge are then seamlessly integrated into our Knowledge Reasoning model, which judiciously navigates through the amalgamated information to arrive at the conclusive answer. Extensive experiments validate the superiority of the proposed method. Compared to previous methods, our method achieves state-of-the-art performance on knowledge-based VQA datasets (65.1% on OK-VQA and 60.1% on A-OKVQA) without resorting to the knowledge produced by GPT-3 (175B).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑