通过图像描述建模视觉语义，提取带注意力的增强型多级跨模态语义不一致表示用于多模态讽刺检测
Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detection
AI总结：
本文提出一种多模态讽刺检测框架，通过引入图像描述作为补充模态，结合跨语言文本特征、自调节残差视觉特征及多级注意力与特征融合，在Twitter和MultiBully数据集上分别达到92.89%和64.48%的最佳准确率。
摘要：
讽刺是一种反讽形式，其特征是字面解释与预期含义之间存在固有的不匹配。尽管文本中的讽刺检测已被广泛研究，但在某些情况下，仅靠文本输入可能不足以感知讽刺。纳入图像等额外上下文线索对于有效识别社交媒体数据中的讽刺至关重要。本研究提出了一种新颖的多模态讽刺检测框架，能够处理输入三元组。这些三元组中的两个组成部分包括输入文本及其关联图像，如数据集中所提供。此外，还以描述性图像说明的形式引入了一种补充模态。纳入这种视觉语义表示的动机是更准确地捕捉文本内容与视觉内容之间的差异，而这些差异是讽刺检测任务的基础。本研究的主要贡献是：（1）一个利用跨语言语言模型的鲁棒文本特征提取分支；（2）一个视觉特征提取分支，其结合了自调节残差卷积网络与轻量级空间感知注意力模块；（3）一种以图像说明形式存在的附加模态，这些说明使用能够读取图像中嵌入文本的编码器-解码器架构生成；（4）不同的注意力模块，用于有效识别文本与两个层级图像表示之间的不一致；（5）通过特征融合实现的多级跨域语义不一致表示。与前沿基线相比，所提出的模型在Twitter多模态讽刺数据集和MultiBully数据集上分别取得了92.89%和64.48%的最佳准确率。
英文摘要：
Sarcasm is a type of irony, characterized by an inherent mismatch between the literal interpretation and the intended connotation. Though sarcasm detection in text has been extensively studied, there are situations in which textual input alone might be insufficient to perceive sarcasm. The inclusion of additional contextual cues, such as images, is essential to recognize sarcasm in social media data effectively. This study presents a novel framework for multimodal sarcasm detection that can process input triplets. Two components of these triplets comprise the input text and its associated image, as provided in the datasets. Additionally, a supplementary modality is introduced in the form of descriptive image captions. The motivation behind incorporating this visual semantic representation is to more accurately capture the discrepancies between the textual and visual content, which are fundamental to the sarcasm detection task. The primary contributions of this study are: (1) a robust textual feature extraction branch that utilizes a cross-lingual language model; (2) a visual feature extraction branch that incorporates a self-regulated residual ConvNet integrated with a lightweight spatially aware attention module; (3) an additional modality in the form of image captions generated using an encoder-decoder architecture capable of reading text embedded in images; (4) distinct attention modules to effectively identify the incongruities between the text and two levels of image representations; (5) multi-level cross-domain semantic incongruity representation achieved through feature fusion. Compared with cutting-edge baselines, the proposed model achieves the best accuracy of 92.89% and 64.48%, respectively, on the Twitter multimodal sarcasm and MultiBully datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
