GeoRef：通过任务形式化、合成监督和基于强化学习的多模态大语言模型解决方案实现几何中的指代表达
GeoRef: Referring Expressions in Geometry via Task Formulation, Synthetic Supervision, and Reinforced MLLM-based Solutions
AI总结：
本文提出几何指代表达理解任务及GeoRef基准，通过合成数据和GRPO微调及验证再生成机制，显著提升多模态模型在几何问题中的定位与推理能力。
摘要：
AI驱动的几何问题求解是一项复杂的视觉语言任务，需要准确的图表解释、数学推理和稳健的跨模态对齐。该任务的一个基础但尚未充分探索的能力是根据自然语言查询识别和解释几何元素。为解决这一问题，我们引入了几何问题中的指代表达理解（REC）任务，该任务评估模型能否根据文本提示在图表中定位点、形状和空间关系。我们提出了GeoRef，一个从现有几何问题语料库构建的基准数据集，具有多样化、高质量的标注和查询。由于该任务缺乏标注数据，我们使用结构化的几何形式语言生成了大规模合成训练数据集，从而广泛覆盖几何概念并促进模型适应。我们探索了两种微调方法：监督微调（SFT）和组相对策略优化（GRPO）。结果表明，GRPO通过更好地将模型行为与任务特定奖励对齐，显著优于SFT。此外，我们提出了一种验证再生成机制，该机制检测错误预测并使用上下文推理历史重新推断答案，进一步提高了准确性。值得注意的是，即使是最先进的多模态大语言模型（MLLM）在此任务上也表现困难，这强调了显式评估和加强几何对齐作为稳健几何问题求解先决条件的必要性。此外，在GeoRef上训练的模型在下游几何推理任务上表现出可衡量的改进，凸显了REC作为多模态数学理解基础更广泛的价值。
英文摘要：
AI-driven geometric problem solving is a complex vision-language task that requires accurate diagram interpretation, mathematical reasoning, and robust cross-modal grounding. A foundational yet underexplored capability for this task is the ability to identify and interpret geometric elements based on natural language queries. To address this, we introduce the task of Referring Expression Comprehension (REC) for geometric problems, which evaluates whether models can localize points, shapes, and spatial relations in diagrams in response to textual prompts. We present GeoRef, a benchmark dataset constructed from existing geometric problem corpora, featuring diverse, high-quality annotations and queries. Due to the lack of annotated data for this task, we generate a large-scale synthetic training dataset using a structured geometric formal language, enabling broad coverage of geometric concepts and facilitating model adaptation. We explore two fine-tuning approaches: Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). Our results show that GRPO significantly outperforms SFT by better aligning model behavior with task-specific rewards. Furthermore, we propose a verify-and-regenerate mechanism that detects incorrect predictions and re-infers answers using contextual reasoning history, further boosting accuracy. Notably, even state-of-the-art Multimodal Large Language Models (MLLMs) struggle with this task, underscoring the necessity of explicitly evaluating and strengthening geometric grounding as a prerequisite for robust geometric problem solving. Moreover, models trained on GeoRef demonstrate measurable improvements on downstream geometric reasoning tasks, highlighting the broader value of REC as a foundation for multimodal mathematical understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
