NavDreamer: 视频模型作为零样本三维导航器
NavDreamer: Video Models as Zero-Shot 3D Navigators
AI总结：
NavDreamer利用视频模型实现零样本三维导航，通过生成视频模型作为语言指令与导航轨迹的接口，结合时空信息和物理动态，实现强大泛化能力。
摘要：
先前的视觉-语言-动作模型在导航方面面临关键限制：数据稀缺、多样，且需要大量劳动力收集，静态表示无法捕捉时间动态和物理定律。我们提出了NavDreamer，一种基于视频的三维导航框架，利用生成视频模型作为语言指令与导航轨迹之间的通用接口。我们的主要假设是，视频编码时空信息和物理动态的能力，结合互联网规模的可用性，能够实现导航中的强大零样本泛化。为缓解生成预测的随机性，我们引入了一种基于采样的优化方法，利用VLM对轨迹进行评分和选择。逆动力学模型被用来从生成的视频计划中解码可执行的航点用于导航。为了系统评估这一范式在几种视频模型骨干网络中的表现，我们引入了一个涵盖物体导航、精确导航、空间定位、语言控制和场景推理的综合基准。大量实验表明，该方法在新物体和未见环境中具有稳健的泛化能力，消融研究揭示导航的高层决策特性使其特别适合于基于视频的规划。
英文摘要：
Previous Vision-Language-Action models face critical limitations in navigation: scarce, diverse data from labor-intensive collection and static representations that fail to capture temporal dynamics and physical laws. We propose NavDreamer, a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and navigation trajectories. Our main hypothesis is that video's ability to encode spatiotemporal information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation. To mitigate the stochasticity of generative predictions, we introduce a sampling-based optimization method that utilizes a VLM for trajectory scoring and selection. An inverse dynamics model is employed to decode executable waypoints from generated video plans for navigation. To systematically evaluate this paradigm in several video model backbones, we introduce a comprehensive benchmark covering object navigation, precise navigation, spatial grounding, language control, and scene reasoning. Extensive experiments demonstrate robust generalization across novel objects and unseen environments, with ablation studies revealing that navigation's high-level decision-making nature makes it particularly suited for video-based planning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
