不看而见：视觉-语言基准测试真的测试视觉吗？
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
AI总结：
通过系统性实验发现，视觉-语言模型对细粒度视觉证据的依赖程度低于预期，表明当前基准测试不足以可靠评估其视觉基础能力。
摘要：
基准测试的准确性通常被隐含地视为反映了视觉-语言模型（VLM）中的基础视觉理解，但尚不清楚这些分数在多大程度上真正反映了对视觉证据的依赖。受一个令人惊讶的观察结果——在广泛使用的幻觉基准测试中，移除大量图像令牌仅轻微降低模型性能——的启发，我们在一组开源VLM中系统地研究了这种不匹配。我们的分析涵盖多个粒度级别，包括全局视觉退化、局部遮挡、问题重述、答案空间扩展以及超出标准准确率的决策级分析。我们进一步用视觉令牌几何的逐层分析补充这些行为结果。在整个实验中，我们发现尽管VLM确实整合了视觉输入，但其预测对细粒度视觉证据丢失的敏感性低于标准准确率所暗示的程度。即使最终预测保持不变，模型对正确答案的内部支持可能已经减弱。我们还补充了表示级分析，显示深层中视觉令牌之间的相似性增加，这为我们的发现提供了一个可能的解释。总之，这些结果表明，当前的基准测试不足以可靠地评估VLM中的细粒度视觉基础。
英文摘要：
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
