城市真实环境中的导航：探索从大规模知识中涌现的导航
City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMs
AI总结：
本文提出稀疏关联视觉导航任务，通过CityNav基准评估MLLMs在真实城市环境中的导航能力，并提出VoP方法提升导航性能。
摘要：
利用多模态大语言模型（MLLMs）开发具身代理，为解决复杂现实任务提供了巨大前景。然而，当前的评估基准仍然主要是语言中心化的或严重依赖于模拟环境，很少探索实际应用中所需的细致、知识密集型推理。为此，我们引入了稀疏关联视觉导航任务，专门设计用于评估MLLMs在具有挑战性和知识密集型的真实环境中的序列决策能力。我们通过CityNav基准来操作化这一任务，该基准涵盖四个多样的全球城市，专门用于评估原始MLLM驱动的代理在城市导航中的能力。代理必须仅依赖视觉输入和内部多模态推理，依次导航50多个决策点，而无需额外的环境注释或专门的架构修改。关键的是，代理必须通过解读城市特定的线索和识别地标来自主实现定位，进行空间推理，并战略性地规划和执行路线以到达目的地。通过广泛的评估，我们证明了当前最先进的MLLMs、推理技术（如GEPA、链式思维、反思）和竞争性基线PReP在这一具有挑战性的设置中表现显著不足。为此，我们提出了路径的口语化（VoP），通过探测城市规模的认知地图（关键地标和目的地方向）来显式地将代理的内部推理进行关联，从而大幅提高导航的成功率。项目网页：https://dwipddalal.github.io/AgentNav/
英文摘要：
Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks. However, current evaluation benchmarks remain predominantly language-centric or heavily reliant on simulated environments, rarely probing the nuanced, knowledge-intensive reasoning essential for practical, real-world scenarios. To bridge this critical gap, we introduce the task of Sparsely Grounded Visual Navigation, explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environment. We operationalize this task with CityNav, a comprehensive benchmark encompassing four diverse global cities, specifically constructed to assess raw MLLM-driven agents in city navigation. Agents are required to rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points without additional environmental annotations or specialized architectural modifications. Crucially, agents must autonomously achieve localization through interpreting city-specific cues and recognizing landmarks, perform spatial reasoning, and strategically plan and execute routes to their destinations. Through extensive evaluations, we demonstrate that current state-of-the-art MLLMs, reasoning techniques (e.g., GEPA, chain-of-thought, reflection) and competitive baseline PReP significantly underperform in this challenging setting. To address this, we propose Verbalization of Path(VoP), which explicitly grounds the agent's internal reasoning by probing city-scale cognitive maps (key landmarks and directions toward the destination) from the MLLM, substantially enhancing navigation success. Project Webpage: https://dwipddalal.github.io/AgentNav/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
