VLN-MME: 诊断MLLMs作为语言引导的视觉导航代理
VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
AI总结：
VLN-MME通过统一评估框架揭示MLLMs在具身导航任务中的局限性，发现其3D空间推理能力不足。
摘要：
多模态大语言模型（MLLMs）在广泛的视觉-语言任务中展现了显著的能力。然而，作为具身代理的表现，这需要多轮对话空间推理和顺序动作预测，仍需进一步探索。我们的工作在视觉-语言导航（VLN）的背景下，引入了一个统一且可扩展的评估框架，通过将传统导航数据集整合到标准化基准中，名为VLN-MME，来探测MLLMs作为零样本代理的表现。我们简化了评估，采用高度模块化且易于访问的设计。这种灵活性简化了实验，使能够在不同的MLLM架构、代理设计和导航任务之间进行结构化比较和组件级消融。关键的是，借助我们的框架，我们发现通过增强我们的基线代理，加入链式推理（CoT）和自我反思，会导致性能的意外下降。这表明MLLMs在具身导航任务中表现出较差的上下文意识；尽管它们可以遵循指令并组织其输出，但其3D空间推理的准确性较低。VLN-MME为系统评估通用MLLMs在具身导航环境中的表现奠定了基础，并揭示了它们在顺序决策能力方面的局限性。我们相信这些发现为MLLMs作为具身代理的后续训练提供了关键指导。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a wide range of vision-language tasks. However, their performance as embodied agents, which requires multi-round dialogue spatial reasoning and sequential action prediction, needs further exploration. Our work investigates this potential in the context of Vision-and-Language Navigation (VLN) by introducing a unified and extensible evaluation framework to probe MLLMs as zero-shot agents by bridging traditional navigation datasets into a standardized benchmark, named VLN-MME. We simplify the evaluation with a highly modular and accessible design. This flexibility streamlines experiments, enabling structured comparisons and component-level ablations across diverse MLLM architectures, agent designs, and navigation tasks. Crucially, enabled by our framework, we observe that enhancing our baseline agent with Chain-of-Thought (CoT) reasoning and self-reflection leads to an unexpected performance decrease. This suggests MLLMs exhibit poor context awareness in embodied navigation tasks; although they can follow instructions and structure their output, their 3D spatial reasoning fidelity is low. VLN-MME lays the groundwork for systematic evaluation of general-purpose MLLMs in embodied navigation settings and reveals limitations in their sequential decision-making capabilities. We believe these findings offer crucial guidance for MLLM post-training as embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
