VLM的眼科检查：指导与检查视觉语言模型的视觉能力
VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models
AI总结：
本文提出一种VLM眼科检查流程及数据集LENS，量化并可视化VLM对颜色、形状及语义匹配的敏感度，发现其对绿色不敏感且形状敏感度取决于LLM容量，为VLM设计与视觉输入预处理提供启发。
摘要：
视觉语言模型（VLMs）在各种基准测试中展现出了有前景的推理能力；然而，我们对它们视觉感知的理解仍然有限。在这项工作中，我们提出了一种眼科检查流程来研究VLM如何感知图像，特别关注视觉识别的关键要素，从基本的颜色和形状到语义层面。为此，我们引入了一个名为LENS的数据集，以指导VLM遵循检查并检查其准备状态。一旦模型准备就绪，我们就进行检查。通过这项检查，我们量化并可视化了VLM对颜色和形状以及语义匹配的敏感度。我们的发现表明，VLM对不同颜色有不同的敏感度，而在不同的VLM中始终对绿色表现出不敏感。此外，我们发现尽管使用相同的固定视觉编码器，但不同的形状敏感度和语义识别取决于LLM的容量。我们的分析和发现有可能启发VLM的设计以及VLM视觉输入的预处理，从而提高应用性能。
英文摘要：
Vision language models (VLMs) have shown promising reasoning capabilities across various benchmarks; however, our understanding of their visual perception remains limited. In this work, we propose an eye examination process to investigate how a VLM perceives images, specifically focusing on key elements of visual recognition, from primitive color and shape to semantic levels. To this end, we introduce a dataset named LENS to guide a VLM to follow the examination and check its readiness. Once the model is ready, we conduct the examination. Through this examination, we quantify and visualize VLMs' sensitivities to color and shape, and semantic matching. Our findings reveal that VLMs have varying sensitivity to different colors while consistently showing insensitivity to green across different VLMs. Also, we found different shape sensitivity and semantic recognition depending on LLM's capacity despite using the same fixed visual encoder. Our analyses and findings have potential to inspire the design of VLMs and the pre-processing of visual input to VLMs for improving application performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
