arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.08006cs.CV
收藏

Dreamland:使用模拟器和生成模型进行可控世界创建

Dreamland: Controllable World Creation with Simulator and Generative Models

University of California, Los Angeles(加州大学洛杉矶分校)

Sicheng Mo, Ziyang Leng, Leon Liu, Weizhen Wang, Honglin He, Bolei Zhou

AI总结:

针对视频生成模型缺乏逐元素可控性的问题,本文提出Dreamland混合框架,通过分层世界抽象连接物理模拟器与预训练生成模型,并构建D3Sim数据集,显著提升图像质量与可控性。

摘要:

大规模视频生成模型能够为动态世界创建合成多样且逼真的视觉内容,但它们通常缺乏逐元素可控性,这阻碍了其在场景编辑和具身AI智能体训练中的应用。我们提出Dreamland,这是一种混合世界生成框架,结合了基于物理的模拟器的细粒度控制与大规模预训练生成模型的照片级真实内容输出。具体而言,我们设计了一种分层世界抽象,将像素级和对象级语义及几何编码为中间表示,以连接模拟器和生成模型。该方法增强了可控性,通过与真实世界分布的早期对齐最小化适配成本,并支持现成使用现有及未来的预训练生成模型。我们进一步构建了D3Sim数据集,以促进混合生成流程的训练和评估。实验表明,Dreamland优于现有基线,图像质量提升50.8%,可控性增强17.9%,并在增强具身智能体训练方面具有巨大潜力。代码和数据将公开提供。

英文摘要:

Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controllability, hindering their use in editing scenes and training embodied AI agents. We propose Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models. In particular, we design a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model. This approach enhances controllability, minimizes adaptation cost through early alignment with real-world distributions, and supports off-the-shelf use of existing and future pretrained generative models. We further construct a D3Sim dataset to facilitate the training and evaluation of hybrid generation pipelines. Experiments demonstrate that Dreamland outperforms existing baselines with 50.8% improved image quality, 17.9% stronger controllability, and has great potential to enhance embodied agent training. Code and data will be made available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑