arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.04992cs.CLcs.AI
收藏

AspectMMKG:具有方面感知实体的多模态知识图谱

AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entities

Fudan University(复旦大学);Soochow University(苏州大学)

Jingdan Zhang, Jiaan Wang, Xiaodan Wang, Zhixu Li, Yanghua Xiao

AI总结:

本文构建了首个具有方面感知图像的多模态知识图谱AspectMMKG,并提出AIR模型用于方面相关图像检索,在实体方面链接任务上取得新最优性能。

摘要:

多模态知识图谱(MMKGs)结合不同模态数据(如文本和图像)以实现对实体的全面理解。尽管近年来大规模多模态知识图谱取得了进展,但现有多模态知识图谱忽视了实体的多方面特性,限制了从不同视角理解实体的能力。本文构建了AspectMMKG,这是第一个具有方面相关图像的多模态知识图谱,通过将图像与不同实体方面进行匹配来实现。具体而言,我们从知识库中收集方面相关图像,并进一步从知识库中提取方面相关句子作为查询,通过在线图像搜索引擎检索大量方面相关图像。最终,AspectMMKG包含2,380个实体、18,139个实体方面和645,383张方面相关图像。我们展示了AspectMMKG在实体方面链接(EAL)下游任务中的可用性,并表明先前的EAL模型在AspectMMKG的帮助下取得了新的最先进性能。为促进方面相关多模态知识图谱的研究,我们进一步提出了方面相关图像检索(AIR)模型,旨在纠正和扩展AspectMMKG中的方面相关图像。我们训练AIR模型,通过融合实体图像、方面和方面图像信息来学习实体图像与实体方面相关图像之间的关系。实验结果表明,AIR模型能够针对给定实体的不同方面检索合适的图像。

英文摘要:

Multi-modal knowledge graphs (MMKGs) combine different modal data (e.g., text and image) for a comprehensive understanding of entities. Despite the recent progress of large-scale MMKGs, existing MMKGs neglect the multi-aspect nature of entities, limiting the ability to comprehend entities from various perspectives. In this paper, we construct AspectMMKG, the first MMKG with aspect-related images by matching images to different entity aspects. Specifically, we collect aspect-related images from a knowledge base, and further extract aspect-related sentences from the knowledge base as queries to retrieve a large number of aspect-related images via an online image search engine. Finally, AspectMMKG contains 2,380 entities, 18,139 entity aspects, and 645,383 aspect-related images. We demonstrate the usability of AspectMMKG in entity aspect linking (EAL) downstream task and show that previous EAL models achieve a new state-of-the-art performance with the help of AspectMMKG. To facilitate the research on aspect-related MMKG, we further propose an aspect-related image retrieval (AIR) model, that aims to correct and expand aspect-related images in AspectMMKG. We train an AIR model to learn the relationship between entity image and entity aspect-related images by incorporating entity image, aspect, and aspect image information. Experimental results indicate that the AIR model could retrieve suitable images for a given entity w.r.t different aspects.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑