arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.08901cs.CV
收藏

基于知识图谱的对比图像预训练

Contrastive Language-Image Pre-Training with Knowledge Graphs

Tsinghua University(清华大学);BNRist(北京国家信息科学与技术研究中心)

Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, Gao Huang

摘要:

近年来,大规模预训练框架得到了快速发展,这些框架可以以统一的形式提取多模态表示,并在转移到下游任务时实现有希望的性能。然而,现有的方法主要集中在简单的图像-文本对的预训练,而忽略了不同模态的概念之间的语义联系。在本文中,我们提出了一个基于知识的预训练框架,称为知识CLIP,它注入语义信息到广泛使用的CLIP模型。通过在预训练过程中引入基于知识的目标,并利用不同类型的知识图谱作为训练数据,该模型可以更好地匹配视觉和语言的语义表示,增强跨场景和模态的推理能力。在各种视觉语言下游任务上的大量实验证明了与原始CLIP和竞争基线相比知识CLIP的有效性。

英文摘要:

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑