视觉-语言模型真的能进行视觉推理吗？一种对模态差距的严格研究
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
AI总结：
本文通过CrossMath基准测试，发现视觉-语言模型在文本输入时表现优异，但加入图像信息后推理性能下降，表明其推理主要依赖文本空间，而非真实视觉证据。
摘要：
视觉-语言模型（VLMs）的推理能力近期受到广泛关注，因其在多种下游任务中的广泛应用。然而，其卓越性能是源于真正的视觉-语言推理，还是主要依赖文本骨干的推理能力仍不明确。为此，我们引入CrossMath，一种新的多模态推理基准，用于受控的跨模态比较。具体而言，我们构建了文本-only、图像-only和图像+文本格式的问题，确保任务相关信息一致，经人类标注员验证。这种严格对齐有效隔离了模态特定的推理差异，同时消除了信息不匹配等混淆因素。对最新VLMs的广泛评估揭示出一致现象：文本推理与视觉推理之间存在显著性能差距。值得注意的是，VLMs在文本-only输入时表现优异，而加入视觉数据（图像+文本）时，性能经常低于文本-only基线。这些发现表明，当前VLMs的推理主要依赖文本空间，对视觉证据的依赖有限。为缓解这一限制，我们整理了一个CrossMath训练集用于VLM微调。实证评估显示，使用此训练集微调可显著提升所有单模态和联合模态的推理性能，并在两个通用视觉推理任务上产生稳健的提升。源代码可在https://github.com/xuyige/CrossMath获取。
英文摘要：
Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
