NYK-MS：一个基于卡通-文字配对数据集的高质量标注多模态隐喻与讽刺理解基准
NYK-MS: A Well-annotated Multi-modal Metaphor and Sarcasm Understanding Benchmark on Cartoon-Caption Dataset
AI总结：
本文提出卡通-文字多模态隐喻与讽刺理解基准 NYK-MS，包含约 1,583/1,578 个样本及 7 类高质量标注任务，并通过零样本和预训练模型实验揭示现有多模态模型在隐喻讽刺理解上的不足与改进方向。
摘要：
隐喻和讽刺是人们交流中常见的修辞表达，尤其是在互联网或青少年流行的网络梗图中。我们创建了一个名为 NYK-MS（NewYorKer for Metaphor and Sarcasm）的新基准，其中包含 1,583 个隐喻理解任务样本和 1,578 个讽刺理解任务样本。这些任务包括：是否包含隐喻/讽刺、哪个词或对象包含隐喻/讽刺、它讽刺了什么以及为什么包含隐喻/讽刺，全部 7 个任务均由至少 3 名标注者进行了高质量标注。我们对数据集进行了多轮标注以提高一致性和质量，并使用 GUI 和 GPT-4V 来提高效率。基于该基准，我们开展了大量实验。在零样本实验中，我们表明大型语言模型（LLM）和大型多模态模型（LMM）无法很好地完成分类任务，并且随着模型规模增大，在其他 5 个任务上的性能有所提升。在传统预训练模型的实验中，我们展示了增强和对齐方法带来的性能提升，这证明我们的基准与以往数据集具有一致性，并要求模型同时理解两种模态。
英文摘要：
Metaphor and sarcasm are common figurative expressions in people's communication, especially on the Internet or the memes popular among teenagers. We create a new benchmark named NYK-MS (NewYorKer for Metaphor and Sarcasm), which contains 1,583 samples for metaphor understanding tasks and 1,578 samples for sarcasm understanding tasks. These tasks include whether it contains metaphor/sarcasm, which word or object contains metaphor/sarcasm, what does it satirize and why does it contains metaphor/sarcasm, all of the 7 tasks are well-annotated by at least 3 annotators. We annotate the dataset for several rounds to improve the consistency and quality, and use GUI and GPT-4V to raise our efficiency. Based on the benchmark, we conduct plenty of experiments. In the zero-shot experiments, we show that Large Language Models (LLM) and Large Multi-modal Models (LMM) can't do classification task well, and as the scale increases, the performance on other 5 tasks improves. In the experiments on traditional pre-train models, we show the enhancement with augment and alignment methods, which prove our benchmark is consistent with previous dataset and requires the model to understand both of the two modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
