MobilityBench:用于评估真实世界移动场景中路径规划智能体的基准
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
AI总结:
提出MobilityBench基准,通过确定性API重放沙箱和多维评估协议,系统评估基于LLM的路径规划智能体,发现现有模型在偏好约束路径规划上表现不佳。
摘要:
由大型语言模型(LLM)驱动的路径规划智能体已成为一种有前景的范式,通过自然语言交互和工具介导的决策支持日常人类移动。然而,在真实世界移动场景中的系统评估受到多样化路由需求、非确定性地图服务和有限可重复性的阻碍。在本研究中,我们引入了MobilityBench,一个用于评估基于LLM的路径规划智能体在真实世界移动场景中的可扩展基准。MobilityBench基于从高德地图收集的大规模匿名真实用户查询构建,覆盖全球多个城市的广泛路径规划意图。为了实现可重复的端到端评估,我们设计了一个确定性API重放沙箱,消除了实时服务带来的环境变化。我们进一步提出了一个以结果有效性为中心的多维评估协议,辅以对指令理解、规划、工具使用和效率的评估。使用MobilityBench,我们在多种真实世界移动场景中评估了多个基于LLM的路径规划智能体,并对其行为和性能进行了深入分析。我们的发现表明,当前模型在基本信息检索和路径规划任务上表现良好,但在偏好约束路径规划上困难重重,突显了在个性化移动应用中仍有显著改进空间。我们在此https URL公开发布基准数据、评估工具包和文档。
英文摘要:
Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making. However, systematic evaluation in real-world mobility settings is hindered by diverse routing demands, non-deterministic mapping services, and limited reproducibility. In this study, we introduce MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. MobilityBench is constructed from large-scale, anonymized real user queries collected from Amap and covers a broad spectrum of route-planning intents across multiple cities worldwide. To enable reproducible, end-to-end evaluation, we design a deterministic API-replay sandbox that eliminates environmental variance from live services. We further propose a multi-dimensional evaluation protocol centered on outcome validity, complemented by assessments of instruction understanding, planning, tool use, and efficiency. Using MobilityBench, we evaluate multiple LLM-based route-planning agents across diverse real-world mobility scenarios and provide an in-depth analysis of their behaviors and performance. Our findings reveal that current models perform competently on Basic information retrieval and Route Planning tasks, yet struggle considerably with Preference-Constrained Route Planning, underscoring significant room for improvement in personalized mobility applications. We publicly release the benchmark data, evaluation toolkit, and documentation at https://github.com/AMAP-ML/MobilityBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。