arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.00316cs.CL
收藏

MapEval:基于地图的基础模型地理空间推理评估

MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models

Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET)(计算机科学与工程系,孟加拉国工程与技术大学);Statistics, Islamic University, Bangladesh(统计学,伊斯兰大学,孟加拉国);Faculty of Information Technology, Monash University, Melbourne, Australia(信息科技学院,墨尔本大学,澳大利亚);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所)

Mahir Labib Dihan, Md Tanvir Hassan, Md Tanvir Parvez, Md Hasebul Hasan, Md Almash Alam, Muhammad Aamir Cheema, Mohammed Eunus Ali, Md Rizwan Parvez

AI总结:

MapEval通过700道覆盖文本、API和视觉推理的多选题评估基础模型的地理空间推理能力,发现30个模型准确率均低于67%,揭示其在距离、方向和路线规划上的关键不足。

摘要:

基础模型的最新进展提升了自主工具使用和推理能力,但它们在基于地图的推理方面的能力仍未得到充分探索。为解决这一问题,我们提出了MapEval,一个旨在通过700道多项选择题评估基础模型在三个不同任务——文本、基于API和视觉推理——上的表现的基准,覆盖180个城市和54个国家,涉及空间关系、导航、旅行规划以及真实世界地图交互。与以往专注于简单位置查询的基准不同,MapEval要求模型处理长上下文推理、API交互和视觉地图分析,使其成为地理空间人工智能最全面的评估框架。在对30个基础模型(包括Claude-3.5-Sonnet、GPT-4o和Gemini-1.5-Pro)的评估中,没有一个模型的准确率超过67%,开源模型的表现明显更差,所有模型均落后人类表现20%以上。这些结果揭示了空间推理中的关键差距,因为模型在距离、方向、路线规划和特定地点推理方面存在困难,凸显了需要更好的地理空间人工智能来弥合基础模型与真实世界导航之间的差距。所有资源可在以下网址获取:https://mapeval.github.io/。

英文摘要:

Recent advancements in foundation models have improved autonomous tool usage and reasoning, but their capabilities in map-based reasoning remain underexplored. To address this, we introduce MapEval, a benchmark designed to assess foundation models across three distinct tasks - textual, API-based, and visual reasoning - through 700 multiple-choice questions spanning 180 cities and 54 countries, covering spatial relationships, navigation, travel planning, and real-world map interactions. Unlike prior benchmarks that focus on simple location queries, MapEval requires models to handle long-context reasoning, API interactions, and visual map analysis, making it the most comprehensive evaluation framework for geospatial AI. On evaluation of 30 foundation models, including Claude-3.5-Sonnet, GPT-4o, and Gemini-1.5-Pro, none surpass 67% accuracy, with open-source models performing significantly worse and all models lagging over 20% behind human performance. These results expose critical gaps in spatial inference, as models struggle with distances, directions, route planning, and place-specific reasoning, highlighting the need for better geospatial AI to bridge the gap between foundation models and real-world navigation. All the resources are available at: https://mapeval.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑