Large-scale video generative models can synthesize diverse and realistic visual content for dynamic world creation, but they often lack element-wise controllability, hindering their use in editing scenes and training embodied AI agents. We propose Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models. In particular, we design a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model. This approach enhances controllability, minimizes adaptation cost through early alignment with real-world distributions, and supports off-the-shelf use of existing and future pretrained generative models. We further construct a D3Sim dataset to facilitate the training and evaluation of hybrid generation pipelines. Experiments demonstrate that Dreamland outperforms existing baselines with 50.8% improved image quality, 17.9% stronger controllability, and has great potential to enhance embodied agent training. Code and data will be made available.
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与移动系统学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);State Key Lab of Intelligent Transportation System, Beihang University(北航智能交通系统国家重点实验室)
SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation
Fudan University, China(复旦大学);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室);Shanghai Jiao Tong University, China(上海交通大学);The Chinese University of Hong Kong, China(香港中文大学);The University of Hong Kong, China(香港大学)
Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images
The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州))
MosaicMem: Hybrid Spatial Memory for Controllable Video World Models
University of Toronto(多伦多大学);Vector Institute(向量研究所);The University of Osaka(大阪大学);Georgia Institute of Technology(佐治亚理工学院);Mujin Inc.(Mujin公司);University of Texas at Austin(德克萨斯大学奥斯汀分校);Taiyuan University of Technology(太原本科技大学)
SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
The School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Waytous Inc.(Waytous公司);The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所)
VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
Westlake University(西湖大学);Zhejiang University(浙江大学);OpenHelix Team(OpenHelix团队);Fudan University(复旦大学);Zhengzhou University(郑州大学);BUPT(北京邮电大学);Hebei University of Technology(河北工业大学)
CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
University of Chinese Academy of Sciences(中国科学院大学);China University of Geosciences (Beijing)(中国地质大学(北京));Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Shandong University(山东大学);University of Science and Technology Beijing(北京科技大学)