先睁眼，再推理：多模态大语言模型中的细粒度视觉数学理解
Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs
AI总结：
针对多模态大语言模型视觉识别误差导致视觉数学推理性能差的问题，提出带几何锚定视觉编码器与特征路由的SVE-Math方法，显著提升了小参数量模型的视觉数学解题表现。
摘要：
当前多模态大语言模型（MLLMs）在需要细粒度视觉理解的数学解题任务上往往表现不佳。这一局限性很大程度上源于图像级对比预训练（如CLIP）过程中对几何基元的感知不足。尽管近期改进数学MLLMs的工作聚焦于扩大数学视觉指令数据集规模以及采用更强的大语言模型（LLM）骨干，但它们往往忽视了视觉识别中持续存在的错误。本文中，我们系统评估了当前最先进MLLMs的视觉 grounding（视觉定位）能力，揭示出视觉定位准确率与解题性能之间存在显著的负相关关系，凸显了细粒度视觉理解的关键作用。值得注意的是，GPT-4o这类先进模型在识别几何实体时的错误率达70%，表明这仍是视觉数学推理的关键瓶颈。为解决该问题，我们提出了一种新方法SVE-Math（选择性视觉增强数学多模态大语言模型），其核心是几何锚定视觉编码器与可动态调整层级视觉特征图贡献的特征路由模块。我们的模型能够识别准确的视觉基元，并生成适配语言模型推理需求的精准视觉提示。实验中，SVE-Math-Qwen2.5-7B在MathVerse上的表现优于其他7B参数量模型15%，在MathVista上的性能与GPT-4V相当。尽管SVE-Math-7B在更小的数据集上训练，但其在GeoQA上取得了具有竞争力的性能，可与在规模大得多的数据集上训练的模型相媲美。我们的研究结果强调了将细粒度视觉理解融入MLLMs的重要性，并为未来研究提供了有前景的方向。
英文摘要：
Current multimodal large language models (MLLMs) often underperform on mathematical problem-solving tasks that require fine-grained visual understanding. The limitation is largely attributable to inadequate perception of geometric primitives during image-level contrastive pre-training (e.g., CLIP). While recent efforts to improve math MLLMs have focused on scaling up mathematical visual instruction datasets and employing stronger LLM backbones, they often overlook persistent errors in visual recognition. In this paper, we systematically evaluate the visual grounding capabilities of state-of-the-art MLLMs and reveal a significant negative correlation between visual grounding accuracy and problem-solving performance, underscoring the critical role of fine-grained visual understanding. Notably, advanced models like GPT-4o exhibit a 70% error rate when identifying geometric entities, highlighting that this remains a key bottleneck in visual mathematical reasoning. To address this, we propose a novel approach, SVE-Math (Selective Vision-Enhanced Mathematical MLLM), featuring a geometric-grounded vision encoder and a feature router that dynamically adjusts the contribution of hierarchical visual feature maps. Our model recognizes accurate visual primitives and generates precise visual prompts tailored to the language model's reasoning needs. In experiments, SVE-Math-Qwen2.5-7B outperforms other 7B models by 15% on MathVerse and is compatible with GPT-4V on MathVista. Despite being trained on smaller datasets, SVE-Math-7B achieves competitive performance on GeoQA, rivaling models trained on significantly larger datasets. Our findings emphasize the importance of incorporating fine-grained visual understanding into MLLMs and provide a promising direction for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
