双眼紧闭？探索多模态LLM的视觉缺陷
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
AI总结：
研究揭示多模态LLM存在系统性视觉缺陷，构建MMVP基准暴露GPT-4V等模型在基本视觉模式上的不足，提出特征混合方法增强视觉基础能力。
摘要：
视觉对语言来说足够好吗？多模态模型的最新进展主要源于大型语言模型（LLM）强大的推理能力。然而，视觉组件通常仅依赖于实例级别的对比语言-图像预训练（CLIP）。我们的研究表明，近期多模态LLM（MLLM）的视觉能力仍存在系统性缺陷。为探究这些错误的根源，我们探索了CLIP的视觉嵌入空间与纯视觉自监督学习之间的差距。我们识别出“CLIP盲对”——即尽管存在明显的视觉差异，但CLIP却感知为相似的图像。利用这些图像对，我们构建了多模态视觉模式（MMVP）基准。MMVP暴露了包括GPT-4V在内的最先进系统在九种基本视觉模式上的简单问题中遇到的困难，它们经常给出错误答案和幻觉解释。我们进一步评估了各种基于CLIP的视觉-语言模型，发现挑战CLIP模型的视觉模式与多模态LLM中有问题的模式之间存在显著相关性。作为解决这些问题的初步努力，我们提出了一种特征混合方法，证明将视觉自监督学习特征与MLLM集成可显著增强其视觉基础能力。综上，我们的研究表明视觉表征学习仍是开放挑战，准确的视觉基础对未来成功的多模态系统至关重要。
英文摘要：
Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
