视觉语言模型能否利用视觉线索解决文本歧义？让视觉双关语来告诉你！
Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!
AI总结：
本文提出UNPIE基准，利用1000个配图双关语设置定位、消歧和重构三项多模态挑战，证明视觉上下文能提升Socratic Models和视觉语言模型解决文本歧义的能力。
摘要：
人类具备多模态素养，能够主动整合来自不同模态的信息以形成推理。面对文本中的词汇歧义等挑战时，我们会借助其他模态进行补充，例如缩略图或教科书插图。机器是否有可能实现类似的多模态理解能力？为此，我们提出了Understanding Pun with Image Explanations（UNPIE），这是一个新颖的基准，旨在评估多模态输入在解决词汇歧义中的作用。双关语因其固有的歧义性而成为这一评估的理想对象。我们的数据集包含1,000个双关语，每个双关语都配有一张解释其双重含义的图像。我们利用标注提出了三个多模态挑战，以评估多模态素养的不同方面：双关语定位（Pun Grounding）、消歧（Disambiguation）和重构（Reconstruction）。结果表明，在给定视觉上下文的情况下，各种Socratic Models和Visual-Language Models相较于纯文本模型均有提升，尤其是在任务复杂度增加时表现更为明显。
英文摘要：
Humans possess multimodal literacy, allowing them to actively integrate information from various modalities to form reasoning. Faced with challenges like lexical ambiguity in text, we supplement this with other modalities, such as thumbnail images or textbook illustrations. Is it possible for machines to achieve a similar multimodal understanding capability? In response, we present Understanding Pun with Image Explanations (UNPIE), a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Puns serve as the ideal subject for this evaluation due to their intrinsic ambiguity. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. We pose three multimodal challenges with the annotations to assess different aspects of multimodal literacy; Pun Grounding, Disambiguation, and Reconstruction. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context, particularly as the complexity of the tasks increases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
