arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.03591cs.AIcs.IR
收藏

结构引导的多模态预训练Transformer用于知识图谱推理

Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning

National University of Defense Technology(国防科技大学)

Ke Liang, Sihang Zhou, Yue Liu, Lingyuan Meng, Meng Liu, Xinwang Liu

AI总结:

本文提出SGMPT,通过图结构编码器和结构引导融合模块将知识图谱结构信息注入文本与视觉特征,是首个用于多模态知识图谱推理的预训练Transformer模型,并在两个基准上取得最优性能。

摘要:

多模态知识图谱(MKGs)以直观方式组织多种模态的信息,能够惠及多个实际下游任务,例如推荐系统和视觉问答。然而,大多数多模态知识图谱仍远未完整,这推动了多模态知识图谱推理模型的蓬勃发展。近年来,随着通用人工智能架构的发展,预训练Transformer模型受到越来越多的关注,尤其是在多模态场景中。然而,用于知识图谱推理(KGR)的多模态预训练Transformer(MPT)研究仍处于早期阶段。作为多模态知识图谱与其他多模态数据之间最大的差异,多模态知识图谱中蕴含的丰富结构信息在现有MPT模型中仍无法被充分利用。大多数现有模型仅将图结构用作检索图,用于匹配与同一实体相连的图像和文本。这种方式阻碍了它们的推理性能。为此,我们提出了图结构引导的多模态预训练Transformer用于知识图谱推理,称为SGMPT。具体而言,采用图结构编码器进行结构特征编码。然后,首次设计了一个具有两种不同策略(即加权求和和对齐约束)的结构引导融合模块,以将结构信息注入文本和视觉特征中。据我们所知,SGMPT是第一个用于多模态知识图谱推理的MPT模型,它挖掘了知识图谱中蕴含的结构信息。在FB15k-237-IMG和WN18-IMG上的大量实验表明,我们的SGMPT优于现有的最先进模型,并证明了所设计策略的有效性。

英文摘要:

Multimodal knowledge graphs (MKGs), which intuitively organize information in various modalities, can benefit multiple practical downstream tasks, such as recommendation systems, and visual question answering. However, most MKGs are still far from complete, which motivates the flourishing of MKG reasoning models. Recently, with the development of general artificial architectures, the pretrained transformer models have drawn increasing attention, especially for multimodal scenarios. However, the research of multimodal pretrained transformer (MPT) for knowledge graph reasoning (KGR) is still at an early stage. As the biggest difference between MKG and other multimodal data, the rich structural information underlying the MKG still cannot be fully leveraged in existing MPT models. Most of them only utilize the graph structure as a retrieval map for matching images and texts connected with the same entity. This manner hinders their reasoning performances. To this end, we propose the graph Structure Guided Multimodal Pretrained Transformer for knowledge graph reasoning, termed SGMPT. Specifically, the graph structure encoder is adopted for structural feature encoding. Then, a structure-guided fusion module with two different strategies, i.e., weighted summation and alignment constraint, is first designed to inject the structural information into both the textual and visual features. To the best of our knowledge, SGMPT is the first MPT model for multimodal KGR, which mines the structural information underlying the knowledge graph. Extensive experiments on FB15k-237-IMG and WN18-IMG, demonstrate that our SGMPT outperforms existing state-of-the-art models, and prove the effectiveness of the designed strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑