基于策略引导的世界模型规划用于语言条件的视觉导航
Policy-Guided World Model Planning for Language-Conditioned Visual Navigation
AI总结：
本文提出PiJEPA框架，结合学习导航策略与潜在世界模型规划，通过微调Octo策略和冻结预训练视觉编码器，提升语言指导下的视觉导航准确性与指令遵循性。
摘要：
在给定自然语言指令下导航至视觉指定目标仍是具身AI的核心挑战。现有方法或依赖反应策略难以进行长周期规划，或使用世界模型在高维空间中行动初始化不佳。本文提出PiJEPA，一个两阶段框架，结合学习导航策略与潜在世界模型规划。第一阶段微调Octo基础通用策略，结合冻结预训练视觉编码器（DINOv2或V-JEPA-2）在CAST导航数据集上进行微调，生成基于当前观察和语言指令的有信息动作分布。第二阶段利用此策略生成的分布作为初始分布，启动基于分离训练的JEPA世界模型的Model Predictive Path Integral（MPPI）规划，预测嵌入空间中的未来潜在状态。通过从策略先验初始化MPPI采样分布而非无信息高斯分布，规划器更快收敛到高质量的动作序列以达到目标。系统研究了视觉编码器主干对策略和世界模型组件的影响。在现实世界导航任务中，PiJEPA显著优于独立策略执行和无指导世界模型规划，实现了更高的目标到达准确性和指令遵循保真度。
英文摘要：
Navigating to a visually specified goal given natural language instructions remains a fundamental challenge in embodied AI. Existing approaches either rely on reactive policies that struggle with long-horizon planning, or employ world models that suffer from poor action initialization in high-dimensional spaces. We present PiJEPA, a two-stage framework that combines the strengths of learned navigation policies with latent world model planning for instruction-conditioned visual navigation. In the first stage, we finetune an Octo-based generalist policy, augmented with a frozen pretrained vision encoder (DINOv2 or V-JEPA-2), on the CAST navigation dataset to produce an informed action distribution conditioned on the current observation and language instruction. In the second stage, we use this policy-derived distribution to warm-start Model Predictive Path Integral (MPPI) planning over a separately trained JEPA world model, which predicts future latent states in the embedding space of the same frozen encoder. By initializing the MPPI sampling distribution from the policy prior rather than from an uninformed Gaussian, our planner converges faster to high-quality action sequences that reach the goal. We systematically study the effect of the vision encoder backbone, comparing DINOv2 and V-JEPA-2, across both the policy and world model components. Experiments on real-world navigation tasks demonstrate that PiJEPA significantly outperforms both standalone policy execution and uninformed world model planning, achieving improved goal-reaching accuracy and instruction-following fidelity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
