多模态UNcommonsense:从奇特到普通和从普通到奇特
Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Odd
AI总结:
多模态UNcommonsense通过R-ICL框架提升模型在非典型场景下的推理能力,实现从奇特到普通和从普通到奇特的转换。
摘要:
多模态情境中的常识推理仍然是人工智能中的基础挑战。我们介绍了多模态UNcommonsense(MUN),一个旨在评估模型处理偏离典型视觉或上下文期望场景能力的基准。MUN将视觉场景与用自然语言描述的意外或不寻常结果配对,促使模型通过日常逻辑来解释看似奇特的图像,或在普通场景中发现意外的解释。为了支持此任务,我们提出了基于检索的上下文学习(R-ICL)框架,该框架将大模型的推理能力转移到小模型上,而无需额外训练。利用一种新的多模态集合检索器(MER),我们的方法即使在图像和文本对刻意不一致时,也能识别出语义相关范例。实验显示,与基线ICL方法相比,平均改进了8.3%,突显了R-ICL在低频、非典型设置中的有效性。MUN为评估和改进视觉-语言模型在现实世界、文化多样性和非典型场景中的鲁棒性和适应性开辟了新方向。
英文摘要:
Commonsense reasoning in multimodal contexts remains a foundational challenge in artificial intelligence. We introduce Multimodal UNcommonsense(MUN), a benchmark designed to evaluate models' ability to handle scenarios that deviate from typical visual or contextual expectations. MUN pairs visual scenes with surprising or unlikely outcomes described in natural language, prompting models to either rationalize seemingly odd images using everyday logic or uncover unexpected interpretations in ordinary scenes. To support this task, we propose a retrieval-based in-context learning (R-ICL) framework that transfers reasoning capabilities from larger models to smaller ones without additional training. Leveraging a novel Multimodal Ensemble Retriever (MER), our method identifies semantically relevant exemplars even when image and text pairs are deliberately discordant. Experiments show an average improvement of 8.3% over baseline ICL methods, highlighting the effectiveness of R-ICL in low-frequency, atypical settings. MUN opens new directions for evaluating and improving visual-language models' robustness and adaptability in real-world, culturally diverse, and non-prototypical scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。