arXiv 2304.139232023-08-06cs.CVcs.CLcs.MM

基于检索的知识增强视觉语言预训练

Retrieval-based Knowledge Augmented Vision Language Pre-training

Sun Yat-sen University（中山大学）；WeChat, Tencent（腾讯微信）

Jiahua Rao, Zifei Shan, Longpo Liu, Yao Zhou, Yuedong Yang

备注：: arXiv admin note: text overlap with arXiv:2210.09338 by other authors

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出REAVL框架，通过知识感知自监督学习建立知识与多模态数据的对应关系并自适应整合有用知识，以极少预训练数据在知识型视觉语言和多模态实体链接任务上取得最优性能。

摘要：

随着大规模视觉与语言表示学习的最新进展，视觉语言预训练（VLP）模型已在各种多模态下游任务上取得了可喜的改进。尽管这些模型功能强大，但尚未充分利用世界知识来发挥其优势。知识增强型VLP的一个关键挑战是知识库与多模态数据之间缺乏明确的联系。此外，并非图像/文本中呈现的所有知识都是有用的，因此先前的方法往往难以有效整合知识、视觉和文本信息。在本研究中，我们提出了REtrieval-based knowledge Augmented Vision Language（REAVL），这是一种新颖的知识增强预训练框架，用于解决上述问题。我们首次引入了一种知识感知的自监督学习方案，该方案能高效地建立知识库与多模态数据之间的对应关系，并识别出有信息量的知识，以改进视觉和文本模态之间对齐与交互的建模。通过自适应地将有信息量的知识与视觉和文本信息相整合，REAVL在基于知识的视觉语言理解和多模态实体链接任务上均取得了新的最先进性能，同时在通用视觉语言任务上也取得了有竞争力的结果，且仅使用了最佳模型0.2%的预训练数据。我们的模型表现出强大的样本效率和有效的知识利用能力。

英文摘要：

With the recent progress in large-scale vision and language representation learning, Vision Language Pre-training (VLP) models have achieved promising improvements on various multi-modal downstream tasks. Albeit powerful, these models have not fully leveraged world knowledge to their advantage. A key challenge of knowledge-augmented VLP is the lack of clear connections between knowledge and multi-modal data. Moreover, not all knowledge present in images/texts is useful, therefore prior approaches often struggle to effectively integrate knowledge, visual, and textual information. In this study, we propose REtrieval-based knowledge Augmented Vision Language (REAVL), a novel knowledge-augmented pre-training framework to address the above issues. For the first time, we introduce a knowledge-aware self-supervised learning scheme that efficiently establishes the correspondence between knowledge and multi-modal data and identifies informative knowledge to improve the modeling of alignment and interactions between visual and textual modalities. By adaptively integrating informative knowledge with visual and textual information, REAVL achieves new state-of-the-art performance uniformly on knowledge-based vision-language understanding and multi-modal entity linking tasks, as well as competitive results on general vision-language tasks while only using 0.2% pre-training data of the best models. Our model shows strong sample efficiency and effective knowledge utilization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。