CounterCount: 一种用于视觉语言模型计数偏差诊断的框架
CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
AI总结：
本文提出CounterCount框架，通过对比事实性与反事实性图像来诊断视觉语言模型在计数任务中的偏差问题，揭示模型对物体级先验知识的依赖，并提出统一的注意力调节策略提升反事实计数准确性。
摘要：
视觉语言模型（VLMs）在多模态推理方面表现出色，但尚不清楚其答案是基于视觉证据还是由学习的语言和世界先验知识驱动。计数提供了一个精确的测试环境：当视觉证据与常识物体知识冲突时，模型必须依赖图像而非典型计数。我们引入CounterCount，一种用于VLMs的反事实计数诊断框架，包含配对的事实性和反事实性图像、编辑过的计数相关属性、验证答案和局部化证据注释。评估最近的VLMs，我们发现其在事实性图像上表现强劲，但在反事实属性变化下持续退化，表明即使存在矛盾的视觉证据，模型仍依赖物体级先验知识。利用局部化注释，我们发现这些失败不仅由于缺失或模糊的视觉证据，而是由于模型对计数相关视觉token的注意力权重不足。我们引入一种统一的推理时间注意力调节策略，重新加权所选的视觉token，使多个VLMs的反事实计数准确率提高高达8%。总体而言，CounterCount揭示了先验驱动的计数失败，并为设计未来的VLMs提供了诊断见解。
英文摘要：
Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
