NL-Eye：面向图像的溯因自然语言推理
NL-Eye: Abductive NLI for Images
AI总结：
本文提出NL-Eye基准，将溯因自然语言推理任务扩展到视觉领域，通过350个三元组示例评估视觉语言模型的视觉溯因推理能力，实验表明现有模型表现接近随机水平，揭示了其在多模态推理上的不足。
摘要：
如果一个基于视觉语言模型（VLM）的机器人在检测到湿地板时，会提醒我们小心滑倒吗？近期的视觉语言模型已展现出令人印象深刻的能力，但它们在推断结果和原因方面的能力仍未得到充分探索。为解决这一问题，我们提出了NL-Eye，一个旨在评估视觉语言模型视觉溯因推理能力的基准。NL-Eye将溯因自然语言推理（NLI）任务适配到视觉领域，要求模型基于前提图像评估假设图像的合理性，并解释其决策。NL-Eye包含350个经过精心整理的三元组示例（共1,050张图像），涵盖多种推理类别：物理、功能、逻辑、情感、文化和社会。数据整理过程包括两个步骤——撰写文本描述并使用文本到图像模型生成图像，这两个步骤都需要大量人工参与，以确保场景具有高质量和挑战性。我们的实验表明，视觉语言模型在NL-Eye上表现显著困难，通常处于随机基线水平，而人类在合理性预测和解释质量方面均表现优异。这表明现代视觉语言模型在溯因推理能力上存在不足。NL-Eye代表着向开发具备稳健多模态推理能力的视觉语言模型迈出的关键一步，可应用于现实世界场景，包括事故预防机器人和生成视频验证。
英文摘要：
Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps - writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
