arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.02445cs.AIcs.LG
收藏

IMKGA-SM:基于序列建模的可解释多模态知识图谱答案预测

IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling

Central South University(中南大学)

Yilin Wen, Biao Luo, Yuqian Zhao

摘要:

多模态知识图谱链接预测旨在提高多模态数据的链接预测任务的准确性和效率。然而,对于复杂的多模态信息和稀疏的训练数据,大多数方法通常很难同时实现可解释性和高精度。针对这一问题,本文提出了一种新的预测模型,即基于序列模型的可解释多模态知识图谱答案预测模型(IMKGA-SM)。首先,提出了多模态细粒度融合方法,并采用Vgg 16和光学字符识别(OCR)技术,有效地从图像和图像中提取文本信息。然后,将知识图谱链接预测任务建模为离线强化学习马尔可夫决策模型,并将其抽象为统一的序列框架。设计了一种交互式的基于感知的奖励期望机制和一种特殊的因果掩蔽机制,将查询“转换”为推理路径;提出了一种自回归动态梯度调整机制,以缓解多模态优化的不足问题。最后,采用两个数据集进行实验,并使用流行的SOTA基线进行比较。实验结果表明,在不同规模的多模态链接预测数据集上,IMKGA-SM的性能优于SOTA基线。

英文摘要:

Multimodal knowledge graph link prediction aims to improve the accuracy and efficiency of link prediction tasks for multimodal data. However, for complex multimodal information and sparse training data, it is usually difficult to achieve interpretability and high accuracy simultaneously for most methods. To address this difficulty, a new model is developed in this paper, namely Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling (IMKGA-SM). First, a multi-modal fine-grained fusion method is proposed, and Vgg16 and Optical Character Recognition (OCR) techniques are adopted to effectively extract text information from images and images. Then, the knowledge graph link prediction task is modelled as an offline reinforcement learning Markov decision model, which is then abstracted into a unified sequence framework. An interactive perception-based reward expectation mechanism and a special causal masking mechanism are designed, which "converts" the query into an inference path. Then, an autoregressive dynamic gradient adjustment mechanism is proposed to alleviate the insufficient problem of multimodal optimization. Finally, two datasets are adopted for experiments, and the popular SOTA baselines are used for comparison. The results show that the developed IMKGA-SM achieves much better performance than SOTA baselines on multimodal link prediction datasets of different sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑