找不到 Waldo：评估视觉语言模型对图像分辨率和细节水平的敏感性
Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level
浏览论文内容
中文总结 AI 辅助
本研究提出受控评估框架和两个指标（AUSC、PVS），系统分析视觉语言模型在高分辨率下的三种失败模式，并为架构设计和数据增强提供改进建议。
中文摘要 AI 辅助
视觉语言模型（VLMs）在各种任务中取得了显著成功，但它们在高分辨率输入上仍存在困难，尤其是当关键信息位于小区域或细节丰富、杂乱场景中时。尽管已有多种方法试图解决这一局限，但缺乏对模型在高分辨率下失败原因的系统性理解。我们引入了一个受控评估框架，通过语义保持变换将分辨率相关的性能退化与任务难度分离开来。我们提出了两个简单指标：缩放曲线下面积（AUSC），用于量化独立于基线准确率的缩放鲁棒性；以及预测方差分数（PVS），用于衡量由分辨率引起的预测不稳定性。通过在5个模型家族和5个基准上的全面实验，我们识别出三种主要失败模式：（1）在视觉令牌限制下因下采样导致的信息丢失，（2）由于补丁边界偏移和非标准宽高比下位置编码脆弱性导致的标记化伪影，以及（3）随着令牌数量增加而出现的注意力稀释。我们的分析表明，即使是先进的模型在处理高分辨率图像时也会遭遇性能下降，且退化模式因架构家族而异。我们为模型架构设计和数据增强策略提供了可操作的见解，以缓解这些局限。
英文摘要
Visual Language Models (VLMs) have achieved remarkable success across diverse tasks, yet they struggle with high-resolution inputs where critical information resides in small regions or detailed, cluttered scenes. While several approaches address this limitation, a systematic understanding of why models fail at high resolutions is lacking. We introduce a controlled evaluation framework that disentangles resolution-related performance degradation from task difficulty through semantics-preserving transformations. We propose two simple metrics: Area Under the Scaling Curve (AUSC), which quantifies scaling robustness independent of baseline accuracy, and Prediction Variance Score (PVS), which measures resolution-induced prediction instability. Through comprehensive experiments across 5 model families and 5 benchmarks, we identify three primary failure modes: (1) information loss from downsampling at vision token limits, (2) tokenization artifacts from patch boundary shifts and positional encoding fragility under non-standard aspect ratios, and (3) attention dilution as token counts increase. Our analysis reveals that even state-of-the-art models suffer from performance drops when processing high-resolution images, with degradation patterns varying systematically by architectural family. We provide actionable insights for model architecture design and data augmentation strategies to mitigate these limitations.
