[解构|重构]视觉语言模型在计数任务中的推理能力
[De|Re]constructing VLMs' Reasoning in Counting
AI总结：
该研究探究视觉语言模型（VLM）在计数任务中推理失效的深层原因，发现其对物体特征、空间排列及干扰物敏感，错误源于末层表征映射偏差，仅微调输出层即可最高提升21%准确率，且在真实数据集上效果一致。
摘要：
视觉语言模型（Vision-Language Models, VLMs）近期因能遵循用户输入指令、在多项下游任务上取得优异表现而受到关注。然而，VLM在视觉推理方面仍存在诸多局限，例如难以识别关系（如空间、时间以及物体间的关系）、理解时间序列（如视频帧）以及对物体计数。在本研究中，我们超越了VLM的分数级基准评估，探究其失效的深层原因，并提出一种针对性方法以提升其推理能力。我们在受控实验条件下，研究了7个最先进的VLM在计数任务中的推理能力。实验表明，VLM对物体的数量与类型、空间排列以及干扰物的共存高度敏感。逐层分析显示，错误源于最后一层表征到输出空间的映射不正确。我们的针对性训练表明，仅微调输出层即可将准确率提升最高达21%。我们通过在真实世界数据集上取得一致的性能提升，验证了这些发现。
英文摘要：
Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. However, VLMs still exhibit several limitations in visual reasoning, such as difficulties in identifying relations (e.g., spatial, temporal, and among objects), understanding temporal sequences (e.g., frames), and counting objects. In this work, we go beyond score-level benchmark evaluations of VLMs by investigating the underlying causes of their failures and proposing a targeted approach to improve their reasoning capabilities. We study the reasoning skills of seven state-of-the-art VLMs in the counting task under controlled experimental conditions. Our experiments show that VLMs are highly sensitive to the number and type of objects, their spatial arrangement, and the co-occurrence of distractors. A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space. Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%. We corroborate these findings by achieving consistent improvements on real-world datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
