视觉语言模型是看见还是猜测？通过措辞控制基准衡量和减少文本先验依赖
Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
AI总结：
本文构建了540张图像的基准，通过为同一图像生成四种措辞变体，衡量视觉语言模型对文本先验的依赖，发现所有模型在最难变体上性能下降，开放模型下降最严重，并通过无图像消融等分析证实了真正的图像依赖。
摘要：
视觉语言模型（VLM）越来越多地被部署在答案必须依据图像内容的场景中，然而它们常常基于文本先验（问题的措辞结合记忆的世界知识）而非图像本身来回答，这夸大了基准分数并产生了自信但无根据的答案。现有基准很少孤立这种行为，因为每张图像通常只与一个固定问题配对。为了衡量这种依赖，我们构建了一个包含540张图像、覆盖六个推理类别的基准，并为相同图像生成四个问题变体，使得措辞而非图像内容成为受控变量。最难的变体直接从图像编写以最小化文本泄漏。我们对十一个VLM进行了基准测试，涵盖从小型开放权重模型到大型闭源系统：每个模型在最难的变体上性能下降，开放模型下降最严重。我们的核心诊断是无图像消融，它将开放权重模型降至其纯文本基线（1%到9%）。进一步的三项分析——LLM评定的难度、低基础到最终文本相似度以及人工重新标注——证实了真正的图像依赖性。与变体构建方式匹配的上下文示例恢复了最高的准确率，而GRPO后训练一个小型VLM在所有四个变体上取得了一致的提升，并泛化到保留的分布外集。文本先验依赖是可测量的，并且部分可通过训练消除。
英文摘要：
Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
