arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09455cs.CVcs.AIcs.CL
收藏

Pandora:迈向具有自然语言动作与视频状态的通用世界模型

Pandora: Towards General World Model with Natural Language Actions and Video States

UC San Diego(加州大学圣迭戈分校);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, Zhengzhong Liu, Eric P. Xing, Zhiting Hu

AI总结:

Pandora通过混合自回归-扩散模型结合预训练LLM与视频模型,实现基于自由文本动作的实时世界状态视频模拟,展示了构建通用世界模型的潜力。

摘要:

世界模型根据不同的动作来模拟世界的未来状态。它们有助于交互式内容创作,并为有根据的长期推理提供基础。当前的基础模型尚未完全满足通用世界模型的能力要求:大语言模型(LLMs)受限于其对语言模态的依赖以及对物理世界的理解有限,而视频模型则缺乏对世界模拟的交互式动作控制。本文通过引入Pandora,向构建通用世界模型迈出了一步。Pandora是一种混合自回归-扩散模型,通过生成视频来模拟世界状态,并允许使用自由文本动作进行实时控制。Pandora通过大规模预训练和指令微调实现了领域通用性、视频一致性和可控性。至关重要的是,Pandora通过整合一个预训练的LLM(7B)和一个预训练的视频模型,绕过了从头训练的高昂成本,仅需额外的轻量级微调。我们展示了Pandora在多种领域(室内/室外、自然/城市、人类/机器人、2D/3D等)中的大量输出结果。结果表明,通过更大规模训练构建更强的通用世界模型具有巨大潜力。

英文摘要:

World models simulate future states of the world in response to different actions. They facilitate interactive content creation and provides a foundation for grounded, long-horizon reasoning. Current foundation models do not fully meet the capabilities of general world models: large language models (LLMs) are constrained by their reliance on language modality and their limited understanding of the physical world, while video models lack interactive action control over the world simulations. This paper makes a step towards building a general world model by introducing Pandora, a hybrid autoregressive-diffusion model that simulates world states by generating videos and allows real-time control with free-text actions. Pandora achieves domain generality, video consistency, and controllability through large-scale pretraining and instruction tuning. Crucially, Pandora bypasses the cost of training-from-scratch by integrating a pretrained LLM (7B) and a pretrained video model, requiring only additional lightweight finetuning. We illustrate extensive outputs by Pandora across diverse domains (indoor/outdoor, natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential of building stronger general world models with larger-scale training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑