arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.16650cs.CLcs.AI
收藏

基于多源语义图的多模态讽刺解释生成

Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

Shandong University(山东大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, Liqiang Nie

AI总结:

针对多模态讽刺解释任务中视觉与语义空间差距及外部知识缺失问题,提出基于多源语义图的TEAM方案,融合对象元数据与ConceptNet知识促进推理,在MORE数据集上优于现有方法。

摘要:

多模态讽刺解释(MuSE)是一项新颖且具挑战性的任务,旨在为多模态社交帖子(图像及其配文)生成自然语言句子,以解释其包含讽刺的原因。尽管现有的先驱研究通过 BART 骨干网络取得了巨大成功,但它忽略了视觉特征空间与解码器语义空间之间的差距、图像的对象级元数据以及潜在的外部知识。为了解决这些局限性,本文提出了一种新颖的基于多源语义图的多模态讽刺解释方案,命名为 TEAM。具体而言,TEAM 从输入图像中提取对象级语义元数据,而非传统的全局视觉特征。同时,TEAM 借助 ConceptNet 获取输入文本和提取出的对象元数据的外部相关知识概念。此后,TEAM 引入了一个多源语义图,全面刻画多源(即配文、对象元数据、外部知识)语义关系,以促进讽刺推理。在公开发布的数据集 MORE 上的大量实验验证了本模型相对于前沿方法的优越性。

英文摘要:

Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm. Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the object-level metadata of the image, as well as the potential external knowledge. To solve these limitations, in this work, we propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta-data. Thereafter, TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑