arXiv 2605.159642026-05-15cs.ROcs.CV

WorldVLN: 用于空域视觉-语言导航的自回归世界动作模型

WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation

Tsinghua University（清华大学）；Shandong University（山东大学）；Manifold AI；Beijing Institute of Technology（北京理工大学）；Northeastern University（东北大学）

Baining Zhao, Jiacheng Xu, Weicheng Feng, Xin Zhang, Zhaolu Wang, Haoyang Wang, Shilong Ji, Ziyou Wang, Jianjie Fang, Zhiheng Zheng, Weichen Zhang, Yu Shang, We… 展开作者

Baining Zhao, Jiacheng Xu, Weicheng Feng, Xin Zhang, Zhaolu Wang, Haoyang Wang, Shilong Ji, Ziyou Wang, Jianjie Fang, Zhiheng Zheng, Weichen Zhang, Yu Shang, Wei Wu, Chen Gao, Xinlei Chen, Yong Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

WorldVLN提出一种自回归世界动作模型，通过预测潜在世界演变并生成可执行的航点动作，提升空域视觉-语言导航性能，优于现有基线模型。

摘要：

空域视觉-语言导航（VLN）要求智能体通过闭环感知与行动在3D环境中遵循自然语言指令。本文认为空域VLN可视为预测驱动的世界-动作问题：智能体应预测潜在世界演变并根据预测后果行动。为此，我们提出WorldVLN，首个针对空域VLN的自回归世界动作模型。不同于生成完整视觉片段的全序列视频生成世界模型，WorldVLN采用潜在自回归视频主干来预测短视界世界状态转换，并直接解码为可执行航点动作。每次动作段执行后，新接收的观测被编码回自回归上下文，实现闭环世界-动作预测。我们进一步引入双阶段训练框架，首先将视频先验在指令条件下的导航动力学中定位，然后开发Action-aware GRPO，首个针对自回归WAMs的强化学习方法，通过下游回放后果优化航点决策。在公开户外和室内基准上，WorldVLN在12%+的成功率提升和挑战性案例中表现更优。它进一步实现零样本迁移至真实无人机部署，表明所提WorldVLN为空间动作任务提供了一条有前景的路径。演示和代码可在https://embodiedcity.github.io/WorldVLN/上获取。

英文摘要：

Aerial vision-language navigation (VLN) requires agents to follow natural-language instructions through closed-loop perception and action in 3D environments. We argue that aerial VLN can be formulated as a prediction-driven world-action problem: the agent should anticipate latent world evolution and act according to the predicted consequences. To this end, we propose WorldVLN, the first autoregressive world action model for aerial VLN. Unlike full-sequence video-generation world models that generate an entire visual clip, WorldVLN adapts a latent autoregressive video backbone to predict short-horizon world-state transitions and directly decodes them into executable waypoint actions. After each action segment is executed, newly received observations are encoded back into the autoregressive context, enabling closed-loop world-action prediction. We further introduce a two-stage training framework that first grounds the video prior in instruction-conditioned navigation dynamics and then develops Action-aware GRPO, the first reinforcement learning method tailored to autoregressive WAMs, to optimize waypoint decisions through their downstream rollout consequences. On public outdoor and indoor benchmarks, WorldVLN consistently outperforms existing Vision-Language-Action baselines with 12\%+ success-rate gains and larger advantages on challenging cases. It further transfers zero-shot to real drone deployment, suggesting that the proposed WorldVLN offers a promising route for spatial action tasks. Demos and code are available at https://embodiedcity.github.io/WorldVLN/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。