arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07973cs.AI
收藏

大型多模态模型距离人类水平的空间行动还有多远?一个面向城市空域目标导向的具身导航基准测试

How Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Northeastern University(东北大学);National University of Defense Technology(国防科技大学);Shandong University(山东大学);BNRist, Tsinghua University(清华大学北京信息科学与技术国家研究中心)

Baining Zhao, Ziyou Wang, Jianjie Fang, Zile Zhou, Yanggang Xu, Yatai Ji, Jiacheng Xu, Qian Zhang, Weichen Zhang, Chen Gao, Xinlei Chen

AI总结:

本文通过构建包含5037个高质量目标导向导航样本的数据集,评估17种代表性模型在城市3D空间中具身空间行动的能力,发现尽管LMMs展现出新兴行动能力,但仍远未达到人类水平,并揭示导航误差在关键决策分歧后迅速发散的现象。

摘要:

大型多模态模型(LMMs)在视觉-语言推理方面表现出强大能力,但其在空间决策和行动方面的能力仍不明确。本文通过一个具有挑战性的场景:在城市3D空间中进行目标导向的导航,探讨LMMs是否能通过具身空间行动达到人类水平。我们首先花费超过500小时构建了一个包含5,037个高质量目标导向导航样本的数据集,重点在于3D垂直动作和丰富的城市语义信息。然后,我们全面评估了17种代表性模型,包括非推理LMMs、推理LMMs、基于代理的方法以及视觉-语言-动作模型。实验表明,当前LMMs展现出新兴行动能力,但仍然远未达到人类水平。此外,我们揭示了一个引人注目的现象:导航错误并不线性累积,而是在关键决策分歧后迅速远离目的地。通过分析这些关键决策分歧处的行为,探讨了LMMs的局限性。最后,我们实验性地探索了四个有前景的改进方向:几何感知、跨视角理解、空间想象和长期记忆。该项目可在:https://github.com/serenditipy-AC/Embodied-Navigation-Bench 上获取。

英文摘要:

Large multimodal models (LMMs) show strong visual-linguistic reasoning but their capacity for spatial decision-making and action remains unclear. In this work, we investigate whether LMMs can achieve embodied spatial action like human through a challenging scenario: goal-oriented navigation in urban 3D spaces. We first spend over 500 hours constructing a dataset comprising 5,037 high-quality goal-oriented navigation samples, with an emphasis on 3D vertical actions and rich urban semantic information. Then, we comprehensively assess 17 representative models, including non-reasoning LMMs, reasoning LMMs, agent-based methods, and vision-language-action models. Experiments show that current LMMs exhibit emerging action capabilities, yet remain far from human-level performance. Furthermore, we reveal an intriguing phenomenon: navigation errors do not accumulate linearly but instead diverge rapidly from the destination after a critical decision bifurcation. The limitations of LMMs are investigated by analyzing their behavior at these critical decision bifurcations. Finally, we experimentally explore four promising directions for improvement: geometric perception, cross-view understanding, spatial imagination, and long-term memory. The project is available at: https://github.com/serenditipy-AC/Embodied-Navigation-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑