我看出你在那儿：大型视觉-语言模型能否理解多模态双关语？
"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?
AI总结：
本文研究了大型视觉-语言模型理解多模态双关语的能力，提出多模态双关语生成流程和MultiPun数据集，通过评估发现模型在区分真实双关语与干扰项上表现不足，并提出提示级和模型级策略提升理解性能。
摘要：
双关语是一种常见的修辞手法，利用多义性和语音相似性来创造幽默。在多模态双关语中，视觉和文本元素协同作用，同时 grounding 文字意义并激发比喻意义。尽管视觉-语言模型（VLMs）广泛用于多模态理解和生成，但其理解双关语的能力尚未被系统研究，这主要是由于缺乏严谨的基准测试。为此，我们首先提出一个多模态双关语生成流程。我们随后引入了MultiPun数据集，该数据集包含多种类型的双关语以及对抗性的非双关语干扰项。我们的评估发现，大多数模型在区分真实双关语与这些干扰项上表现不足。此外，我们提出了提示级和模型级的策略以提升双关语的理解能力，平均在F1分数上提升了16.5%。我们的发现为开发未来能够通过跨模态推理掌握人类样幽默的VLMs提供了有价值的见解。
英文摘要：
Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
