基于语言的视觉导航世界建模
Language-Conditioned World Modeling for Visual Navigation
AI总结：
本文研究了基于语言的视觉导航问题，提出LCVN数据集并开发了两种模型家族，通过语言 grounding、未来状态预测和动作生成实现统一任务学习。
摘要：
我们研究了语言条件的视觉导航（LCVN），其中一个具身代理仅根据初始自体视觉观察就需遵循自然语言指令。在没有目标图像的情况下，代理必须依靠语言来塑造感知和连续控制，使grounding问题特别具有挑战性。我们将此问题建模为开放环路轨迹预测，条件于语言指令，并引入LCVN数据集，包含39,016条轨迹和117,048条经人类验证的指令，支持在多种环境和指令风格中进行可重复研究。利用此数据集，我们开发了LCVN框架，通过两个互补的模型家族将语言 grounding、未来状态预测和动作生成联系起来。第一类结合LCVN-WM，一种基于扩散的世界模型，与LCVN-AC，一种在世界模型潜在空间中训练的actor-critic代理。第二类LCVN-Uni采用自回归多模态架构，可预测动作和未来观察。实验表明，这些家族各有优势：前者提供更时间上连贯的轨迹，而后者在未见环境中泛化更好。总体来看，这些观察表明在统一任务设置中联合研究语言 grounding、想象和策略学习的价值，LCVN为进一步研究语言条件的世界模型提供了具体基础。代码可在https://github.com/F1y1113/LCVN获取。
英文摘要：
Goal-conditioned visual navigation has been a long-standing testbed for embodied AI. We study a natural language-conditioned variant, language-conditioned visual navigation (LCVN), in which an embodied agent must follow a natural language instruction given only an initial egocentric observation. Without access to goal images, the agent must rely on language to shape its perception and continuous control. We introduce the LCVN Dataset, a benchmark of 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles. Building on this benchmark, we study two complementary paradigms: (i) latent-imagination policy learning, in which a diffusion-based world model (LCVN-WM) imagines future observations and an actor-critic agent (LCVN-AC) learns its policy entirely within the imagined latent space; and (ii) unified autoregressive prediction, in which a single multimodal backbone (LCVN-Uni) jointly predicts actions and observations in one forward pass over a shared token sequence. Experiments show that two paradigms offer complementary strengths: latent imagination produces more temporally coherent rollouts, whereas unified prediction generalizes better to unseen environments. Targeted ablations further isolate the contributions of language guidance, conditioning signals, and instruction style, clarifying when language grounding versus dynamics modeling is the performance bottleneck. Together, these findings position LCVN as a testbed for studying how language, imagination, and decision-making interact in embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
