arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.15445cs.CLcs.AIcs.CV
收藏

IRFL:比喻性语言的图像识别

IRFL: Image Recognition of Figurative Language

The Hebrew University of Jerusalem(希伯来大学)

Ron Yosef, Yonatan Bitton, Dafna Shahaf

AI总结:

本文开发了IRFL多模态比喻语言理解数据集与基准,引入两个新任务,实验表明当前先进视觉语言模型表现远低于人类。

摘要:

诸如暗喻、明喻和习语等修辞手法是人类交流的重要组成部分。它们在多种形式的语篇中无处不在,使人们能够传达复杂、抽象的思想并唤起情感。由于比喻形式通常通过多种模态(例如文本和图像)传达,理解多模态比喻语言是一项重要的AI挑战,它交织了深刻的视觉、语言、常识和文化知识。在这项工作中,我们开发了Image Recognition of Figurative Language (IRFL)数据集。我们利用人工标注和我们创建的自动流程来生成多模态数据集,并引入两个新任务作为多模态比喻语言理解的基准。我们使用最先进的视觉和语言模型进行了实验,发现表现最好的模型(22%)的表现远低于人类(97%)。我们发布我们的数据集、基准和代码,希望能推动能更好理解比喻语言的模型的发展。

英文摘要:

Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code, in hopes of driving the development of models that can better understand figurative language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑