arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.04447cs.CVcs.RO
收藏

DreamVLA:融合全面世界知识的视觉-语言-动作模型

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

SJTU(上海交通大学);EIT(欧洲研究所);THU(清华大学);Galbot;PKU(北京大学);UIUC(伊利诺伊大学香槟分校);USTC(中国科学技术大学)

Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, Fan Lu, He Wang, Zhizheng Zhang, Li Yi, Wenjun Zeng, Xin Jin

AI总结:

针对现有视觉-语言-动作模型在图像预测中存在冗余信息且缺乏全面世界知识的局限,提出DreamVLA框架,通过整合世界知识预测实现逆动力学建模,提升机器人操作任务的泛化性与推理能力。

摘要:

视觉-语言-动作(VLA)模型的最新进展在整合图像生成与动作预测以提升机器人操作的泛化性和推理能力方面展现出潜力。然而,现有方法局限于挑战性的基于图像的预测,该预测存在冗余信息且缺乏全面关键的世界知识,包括动态、空间和语义信息。为解决这些局限,我们提出DreamVLA,一种新颖的VLA框架,通过整合全面的世界知识预测实现逆动力学建模,从而为操作任务建立感知-预测-动作循环。具体而言,DreamVLA引入动态区域引导的世界知识预测,并结合空间和语义线索,为动作规划提供紧凑且全面的表示。这种设计与人类与世界交互的方式一致:行动前先形成抽象的多模态推理链。为减轻训练中动态、空间和语义信息间的干扰,我们采用块结构注意力机制,屏蔽它们的相互注意力,防止信息泄漏并保持各表示清晰且解耦。此外,为建模未来动作的条件分布,我们使用基于扩散的Transformer,将动作表示与共享潜在特征解耦。在真实世界和仿真环境中的大量实验表明,DreamVLA在真实机器人任务上达到76.7%的成功率,在CALVIN ABC-D基准上达到4.44的平均长度。

英文摘要:

Recent advances in vision-language-action (VLA) models have shown promise in integrating image generation with action prediction to improve generalization and reasoning in robot manipulation. However, existing methods are limited to challenging image-based forecasting, which suffers from redundant information and lacks comprehensive and critical world knowledge, including dynamic, spatial and semantic information. To address these limitations, we propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling, thereby establishing a perception-prediction-action loop for manipulation tasks. Specifically, DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning. This design aligns with how humans interact with the world by first forming abstract multimodal reasoning chains before acting. To mitigate interference among the dynamic, spatial and semantic information during training, we adopt a block-wise structured attention mechanism that masks their mutual attention, preventing information leakage and keeping each representation clean and disentangled. Moreover, to model the conditional distribution over future actions, we employ a diffusion-based transformer that disentangles action representations from shared latent features. Extensive experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks and 4.44 average length on the CALVIN ABC-D benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑