arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01934cs.CL
收藏

多模态实体链接的最优传输引导相关性分配

Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)

Zefeng Zhang, Jiawei Sheng, Chuang Zhang, Yunzhi Liang, Wenyuan Zhang, Siqi Wang, Tingwen Liu

AI总结:

针对多模态实体链接中注意力机制过度集中于局部相关性的问题,提出最优传输引导的OT-MEL框架,利用多模态特征与指称-实体相关性增强融合与细粒度匹配,并通过知识蒸馏加速预测,显著超越现有基线。

摘要:

多模态实体链接(MEL)旨在将多模态上下文中的歧义指称链接到多模态知识图谱中的实体。一个关键挑战是充分利用指称与实体之间的多元素相关性,以弥合模态鸿沟并实现细粒度的语义匹配。现有方法尝试了多种局部相关机制,严重依赖自动学习的注意力权重,这可能导致过度集中于部分相关性。为缓解此问题,我们将相关性分配问题表述为最优传输(OT)问题,并提出了一种新型MEL框架,即OT-MEL,采用OT引导的相关性分配。由此,我们利用多模态特征之间的相关性来增强多模态融合,并利用指称与实体之间的相关性来增强细粒度匹配。为加速模型预测,我们进一步利用知识蒸馏将OT分配知识转移到注意力机制中。实验结果表明,我们的模型显著优于先前的最先进基线,并证实了OT引导相关性分配的有效性。

英文摘要:

Multimodal Entity Linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multi-element correlations between mentions and entities to bridge modality gap and enable fine-grained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OT-MEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑