arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.11751cs.ROcs.AI
收藏

基于语义泛化的世界模型用于规划

Grounded World Model: Latent Planning with Language Goals

Independent(独立);EPFL(瑞士联邦理工学院洛桑);Beihang University(北京航空航天大学);University of Toronto(多伦多大学);NUS(新加坡国立大学)

Quanyi Li, Lan Feng, Haonan Zhang, Wuyang Li, Letian Wang, Alexandre Alahi, Harold Soh

AI总结:

本文提出基于视觉-语言对齐潜在空间的世界模型,用于改进视觉-运动模型控制,实现更广泛的语义泛化能力。

摘要:

在模型预测控制(MPC)中,世界模型预测各种动作提案的未来结果,然后通过评分来指导最佳动作的选择。对于视觉-运动MPC,评分函数是预测图像与目标图像在预训练视觉编码器(如DINO和JEPA)潜在空间中的距离度量。然而,在任务执行前获取目标图像具有挑战性,特别是在新环境中。此外,通过图像传达目标相比自然语言交互性有限。本文提出在视觉-语言对齐的潜在空间中学习一个接地世界模型(GWM)。结果,每个提出的动作根据其未来结果与任务指令的接近程度进行评分,通过嵌入相似性反映。这种方法将视觉-运动MPC转变为VLA,超越基于VLM的VLA在语义泛化方面。在提出的WISER基准上,GWM-MPC在包含288个未见过视觉信号和指代表达的任务测试集上达到87%的成功率,尽管在训练期间演示了动作。相比之下,传统VLA在测试集上的平均成功率为22%,尽管它们在训练集上过度拟合,成功率为90%。

英文摘要:

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑