通过视觉实体增强和多粒度图像噪声过滤迈向更好的多模态关键短语生成
Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering
AI总结:
提出一种新颖的多模态关键短语生成模型,通过引入外部视觉实体增强输入并利用多粒度图像噪声过滤机制,有效解决图文不匹配和辅助信息不足问题,在基准数据集上达到SOTA性能。
摘要:
多模态关键短语生成旨在生成一组能够代表输入文本-图像对核心要点的关键短语。在此方面,主流方法主要关注用于关键短语生成的多模态融合。然而,仍存在两个主要缺陷:1)只有有限数量的来源(例如图像描述)可以被用来提供辅助信息。然而,它们可能不足以支持后续的关键短语生成。2)输入文本和图像通常并不完全匹配,因此图像可能会向模型引入噪声。为了解决这些局限性,在本文中,我们提出了一种新颖的多模态关键短语生成模型,该模型不仅利用外部知识丰富了模型输入,还有效地过滤了图像噪声。首先,我们将图像的外部视觉实体作为模型的补充输入引入,这有利于关键短语生成的跨模态语义对齐。其次,我们同时计算图像-文本匹配分数和图像区域-文本相关性分数,以执行多粒度图像噪声过滤。特别地,我们引入图像区域与真实关键短语之间的相关性分数,以改进上述相关性分数的计算。为了证明我们模型的有效性,我们在基准数据集上进行了多组实验。实验结果和深入分析表明,我们的模型达到了最先进的性能。我们的代码可在 https://github.com/DeepLearnXMU/MM-MKP 获取。
英文摘要:
Multi-modal keyphrase generation aims to produce a set of keyphrases that represent the core points of the input text-image pair. In this regard, dominant methods mainly focus on multi-modal fusion for keyphrase generation. Nevertheless, there are still two main drawbacks: 1) only a limited number of sources, such as image captions, can be utilized to provide auxiliary information. However, they may not be sufficient for the subsequent keyphrase generation. 2) the input text and image are often not perfectly matched, and thus the image may introduce noise into the model. To address these limitations, in this paper, we propose a novel multi-modal keyphrase generation model, which not only enriches the model input with external knowledge, but also effectively filters image noise. First, we introduce external visual entities of the image as the supplementary input to the model, which benefits the cross-modal semantic alignment for keyphrase generation. Second, we simultaneously calculate an image-text matching score and image region-text correlation scores to perform multi-granularity image noise filtering. Particularly, we introduce the correlation scores between image regions and ground-truth keyphrases to refine the calculation of the previously-mentioned correlation scores. To demonstrate the effectiveness of our model, we conduct several groups of experiments on the benchmark dataset. Experimental results and in-depth analyses show that our model achieves the state-of-the-art performance. Our code is available on https://github.com/DeepLearnXMU/MM-MKP.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。