上下文学习可帮助视觉-语言模型克服训练先验
In-Context Learning Can Help Vision-Language Models Overcome Training Priors
AI总结：
本研究提出用受控视觉上下文学习引导视觉-语言模型聚焦视觉证据，可修正其因训练先验导致的错误，证实模型具备标准评估未发现的视觉 grounding 能力，且该方法可推广至多类模型。
摘要：
视觉-语言模型（VLMs）在训练过程中会学习到强统计规律，当输入图像违背这些规律时，模型可能无法对视觉证据做出恰当响应。要区分这类由先验导致的错误与模型缺失基础视觉能力导致的错误颇具挑战性。本研究采用受控视觉上下文学习，引导模型聚焦视觉证据，证明仅这一操作就常足以修正错误，表明模型实际上具备相关视觉能力。具体而言，在5项视觉中心任务中，演示对与先验冲突的反事实示例性能提升显著：Qwen3-VL 235B提升27.2%，Qwen3-VL 32B提升15.1%，同时对与先验一致的示例准确率几乎无影响。这些发现可推广至5个VLM家族，提升幅度为9.7%至30.3%。我们分析视觉表征发现，演示确实会将注意力重定向至 grounding 线索，同时减少对支持已学先验线索的依赖。最后，我们发现当基础视觉能力较弱或缺失时，上下文学习的作用有限。综上，这些结果表明当前VLMs具备标准评估可能忽略的视觉 grounding 能力，凸显了在诊断和提升模型能力时需采用受控评估设计的必要性。
英文摘要：
Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples ($+27.2\%$ for Qwen3-VL 235B and $+15.1\%$ for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from $+9.7\%$ to $+30.3\%$. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
