Grid2Matrix: 揭示视觉语言模型中的数字失读
Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
AI总结：
Grid2Matrix通过控制视觉复杂度测试视觉语言模型在捕捉细节方面的不足，发现模型在小网格上表现不佳，揭示了视觉编码与语言表达间的差距，即'数字失读'。
摘要：
视觉语言模型（VLMs）在许多多模态推理基准上表现优异，但这些评估通常不需要对图像进行详尽读取，因此可能掩盖了未能忠实捕捉所有视觉细节的失败。我们引入Grid2Matrix（G2M），一种受控基准，模型需根据颜色网格和颜色到数字的映射输出对应矩阵。通过改变网格大小和颜色数量，G2M提供了一种增加视觉复杂度同时最小化语义混淆的方法。我们发现VLMs在零样本端到端评估中表现出急剧的早期崩溃，失败于 surprisingly 小的网格，而非随着任务密度增加而逐渐退化。我们探测了VLMs中两个代表性的家族的视觉编码器，发现它们保留了比相应端到端输出更多的网格信息。这表明失败不仅仅由视觉编码本身解释，还反映了从视觉特征中可恢复的内容与最终语言表达之间的差距。我们称这种差距为\textit{数字失读}。进一步分析显示，这些错误高度结构化，且强烈依赖于网格单元与视觉补丁边界重叠的方式。我们还发现，常见的策略如模型扩展和多模态对齐无法完全消除这种失败模式。我们期望G2M能作为理解VLMs在何处及如何丢失细视觉细节的有用测试平台，并用于评估那些缺失甚至小视觉细节可能重要的任务，如表格、图表、表单和GUIs。
英文摘要：
Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the image and can therefore obscure failures in faithfully capturing all visual details. We introduce Grid2Matrix (G2M), a controlled benchmark in which a model is shown a color grid and a color-to-number mapping, and must output the corresponding matrix. By varying grid size and the number of colors, G2M provides a simple way to increase visual complexity while minimizing semantic confounds. We find that VLMs exhibit a sharp early collapse in zero-shot end-to-end evaluation, failing on surprisingly small grids rather than degrading gradually as the task becomes denser. We probe the visual encoders of VLMs from two representative families and find that they preserve substantially more of the grid information than the corresponding end-to-end outputs. This suggests that the failure is not explained by visual encoding alone, but also reflects a gap between what remains recoverable from visual features and what is ultimately expressed in language. We term this gap \textit{Digital Agnosia}. Further analyses show that these errors are highly structured and depend strongly on how grid cells overlap with visual patch boundaries. We also find that common strategies such as model scaling and multimodal alignment do not fully eliminate this failure mode. We expect G2M to serve as a useful testbed for understanding where and how VLMs lose fine visual details, and for evaluating tasks where missing even small visual details can matter, such as tables, charts, forms, and GUIs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
