arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.04468cs.CL
收藏

用于少样本多模态讽刺检测的双模态感知门控提示调优

Dual Modality-Aware Gated Prompt Tuning for Few-Shot Multimodal Sarcasm Detection

IIT Guwahati(印度果阿瓦提理工学院)

Soumyadeep Jana, Abhrajyoti Kundu, Sanasam Ranbir Singh

AI总结:

本文提出DMDP框架,通过门控模态特定深度提示、跨层提示共享和跨模态对齐,提升少样本多模态讽刺检测性能,并在多个数据集上优于基线方法。

摘要:

社交媒体上多模态内容的广泛使用,提高了对有效讽刺检测以改善观点挖掘的需求。然而,现有模型严重依赖大规模标注数据集,使其不太适用于标注数据稀缺的现实场景。这促使我们需要在少样本设定下探索该问题。为此,我们提出了DMDP(Deep Modality-Disentangled Prompt Tuning,深度模态解耦提示调优),一个用于少样本多模态讽刺检测的新框架。与先前在不同模态上使用浅层统一提示的方法不同,DMDP为文本编码器和视觉编码器采用门控的、模态特定的深度提示。这些提示被注入到多个层中,以实现层次化特征学习,并更好地捕捉多样化的讽刺类型。为增强模态内学习,我们引入了一种跨层提示共享机制,使模型能够聚合低层和高层语义线索。此外,一个跨模态提示对齐模块使图像和文本表示之间能够进行细致交互,从而提升模型检测细微讽刺意图的能力。在两个公开数据集上的实验表明,DMDP在少样本和极低资源设定下均具有优越性能。进一步的跨数据集评估显示,DMDP在不同领域之间具有良好的泛化能力,并持续优于基线方法。

英文摘要:

The widespread use of multimodal content on social media has heightened the need for effective sarcasm detection to improve opinion mining. However, existing models rely heavily on large annotated datasets, making them less suitable for real-world scenarios where labeled data is scarce. This motivates the need to explore the problem in a few-shot setting. To this end, we introduce DMDP (Deep Modality-Disentangled Prompt Tuning), a novel framework for few-shot multimodal sarcasm detection. Unlike prior methods that use shallow, unified prompts across modalities, DMDP employs gated, modality-specific deep prompts for text and visual encoders. These prompts are injected across multiple layers to enable hierarchical feature learning and better capture diverse sarcasm types. To enhance intra-modal learning, we incorporate a prompt-sharing mechanism across layers, allowing the model to aggregate both low-level and high-level semantic cues. Additionally, a cross-modal prompt alignment module enables nuanced interactions between image and text representations, improving the model's ability to detect subtle sarcastic intent. Experiments on two public datasets demonstrate DMDP's superior performance in both few-shot and extremely low-resource settings. Further cross-dataset evaluations show that DMDP generalizes well across domains, consistently outperforming baseline methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑