Response Wide Shut：基础视觉语言模型能力中的惊人发现
Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
AI总结：
该研究通过构建基础视觉任务测试集，对比视觉编码器特征探针、中间视觉-语言投影层与VLM的性能，揭示了当前最先进VLM的响应缺陷，为后续模型优化提供依据。
摘要：
视觉语言模型（Vision-Language Models, VLMs）已成为解决各类复杂计算机视觉问题的通用工具。这类模型已被证明具备强大能力，但同时也缺失一些基础的视觉理解技能。本文旨在通过构建一系列测试，探究当前最先进（SoTA）VLM在基础视觉任务上的局限性，具体包括目标分类、空间排列理解以及（通过计数实现的）单个目标实例勾勒能力，以明确究竟是哪些设计组件可能存在不足。重要的是，我们的研究远超现有基准——现有基准仅衡量VLM的最终性能，而我们还将其与直接基于视觉编码器提取的特征（图像嵌入）训练的探针性能，以及许多SoTA模型（如LLaVA、BLIP、InstructBLIP）中用于衔接图像编码器与大语言模型解码器输出的中间视觉-语言投影层的性能进行了对比分析。通过这些工作，我们揭示了VLM响应中尚未被充分关注的缺陷，并得出了若干重要结论，可为未来训练和开发更高效的VLM模型提供帮助。
英文摘要：
Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, also lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks: object classification, understanding spatial arrangement, and ability to delineate individual object instances (through counting), by constructing a series of tests that probe which components of design, specifically, maybe lacking. Importantly, we go significantly beyond the current benchmarks, that simply measure final performance of VLM, by also comparing and contrasting it to performance of probes trained directly on features obtained from visual encoder (image embeddings), as well as intermediate vision-language projection used to bridge image-encoder and LLM-decoder ouput in many SoTA models (e.g., LLaVA, BLIP, InstructBLIP). In doing so, we uncover nascent shortcomings in VLMs response and make a number of important observations which could help train and develop more effective VLM models in future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
