FRIEDA：评估视觉语言模型中多步骤制图推理的基准
FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Models
AI总结：
FRIEDA 是一个评估视觉语言模型多步骤制图推理能力的基准，通过多步骤推理和跨地图定位测试，揭示了当前模型在空间智能方面的不足。
摘要：
制图推理是指通过对齐图例、地图比例、方向、地图文本和几何图形来解读地理关系的能力。尽管作为具体认知能力对于灾难响应和城市规划等关键任务至关重要，但这种能力仍然很少被评估。基于图表和信息图理解的进步，最近关于地图视觉问答（VQA）的大型视觉语言模型研究往往将地图视为图表的特殊类型。相比之下，地图VQA要求理解分层符号（如符号、几何图形和文本标签）以及与方向和距离相关的空间关系，这些关系通常跨越多个地图，并且无法通过图表式评估来捕捉。为了解决这一差距，我们引入了FRIEDA，一个用于测试大型视觉语言模型（LVLMs）中复杂开放性制图推理的基准。FRIEDA从不同领域和地理区域的文档和报告中获取真实地图图像。遵循地理信息系统（GIS）文献中的分类，FRIEDA针对三种空间关系：拓扑关系（边界、相等、相交、包含）、度量关系（距离）和方向关系（方向）。所有问题都需要多步骤推理，许多问题需要跨地图定位和推理。我们评估了 eleven 个最先进的 LVLMs，在两种设置下：（1）直接设置，即我们提供与问题相关的地图；（2）上下文设置，即模型可能需要在推理前识别与问题相关地图。即使是最强的模型，Gemini-2.5-Pro 和 GPT-5-Think，也仅分别达到 38.20% 和 37.20% 的准确率，远低于人类 84.87% 的表现。这些结果揭示了多步骤制图推理的持续差距，使 FRIEDA 成为推动 LVLMs 空间智能进步的严格基准。
英文摘要：
Cartographic reasoning is the skill of interpreting geographic relationships by aligning legends, map scales, compass directions, map texts, and geometries across one or more map images. Although essential as a concrete cognitive capability and for critical tasks such as disaster response and urban planning, it remains largely unevaluated. Building on progress in chart and infographic understanding, recent large vision language model studies on map visual question-answering often treat maps as a special case of charts. In contrast, map VQA demands comprehension of layered symbology (e.g., symbols, geometries, and text labels) as well as spatial relations tied to orientation and distance that often span multiple maps and are not captured by chart-style evaluations. To address this gap, we introduce FRIEDA, a benchmark for testing complex open-ended cartographic reasoning in LVLMs. FRIEDA sources real map images from documents and reports in various domains and geographical areas. Following classifications in Geographic Information System (GIS) literature, FRIEDA targets all three categories of spatial relations: topological (border, equal, intersect, within), metric (distance), and directional (orientation). All questions require multi-step inference, and many require cross-map grounding and reasoning. We evaluate eleven state-of-the-art LVLMs under two settings: (1) the direct setting, where we provide the maps relevant to the question, and (2) the contextual setting, where the model may have to identify the maps relevant to the question before reasoning. Even the strongest models, Gemini-2.5-Pro and GPT-5-Think, achieve only 38.20% and 37.20% accuracy, respectively, far below human performance of 84.87%. These results reveal a persistent gap in multi-step cartographic reasoning, positioning FRIEDA as a rigorous benchmark to drive progress on spatial intelligence in LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
