arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.11965cs.CV
收藏

UrbanWorld:面向三维城市生成的城市场景世界模型

UrbanWorld: An Urban World Model for 3D City Generation

Tsinghua University(清华大学);Tsingroc Inc.(清睿科技公司);University of Science and Technology Beijing(北京科技大学)

Yu Shang, Yuming Lin, Yu Zheng, Hangyu Fan, Jingtao Ding, Jie Feng, Jiansheng Chen, Li Tian, Yong Li

AI总结:

本文提出首个生成式城市场景世界模型UrbanWorld,通过布局生成、MLLM场景设计、渐进式三维扩散渲染和场景细化四阶段自动创建逼真可交互的三维城市,并在视觉指标和智能体交互上验证其有效性。

摘要:

城市作为人类生活的基本环境,包含建筑、道路和植被等多种物理要素,并与行人、车辆等动态实体持续交互。构建逼真、可交互的三维城市环境对于培育通用人工智能(AGI)系统以及构建能够在真实环境中像人类一样感知、决策和行动的AI智能体至关重要。然而,创建高保真三维城市环境通常需要设计人员大量手工劳动,涉及复杂城市元素的精细细节刻画与表示。因此,自动完成这一任务仍是一项长期挑战。针对该问题,我们提出UrbanWorld,这是首个生成式城市场景世界模型,能够通过灵活的控制条件自动创建定制化、逼真且可交互的三维城市场景。UrbanWorld在生成流程中包含四个关键阶段:基于OSM数据或带有语义图和高度图的城市布局进行灵活的三维布局生成,利用Urban MLLM进行城市场景设计,通过渐进式三维扩散实现可控的城市资产渲染,以及借助MLLM辅助的场景细化。我们在五个视觉指标上进行了广泛的定量分析,证明UrbanWorld达到了SOTA的生成真实感。随后,我们给出了关于UrbanWorld使用文本和基于图像的提示进行可控生成能力的定性结果。最后,我们通过展示所创建环境中的智能体感知与导航,验证了这些环境的可交互性。我们将UrbanWorld作为开源工具贡献,可在https://github.com/Urban-World/UrbanWorld获取。

英文摘要:

Cities, as the essential environment of human life, encompass diverse physical elements such as buildings, roads and vegetation, which continuously interact with dynamic entities like people and vehicles. Crafting realistic, interactive 3D urban environments is essential for nurturing AGI systems and constructing AI agents capable of perceiving, decision-making, and acting like humans in real-world environments. However, creating high-fidelity 3D urban environments usually entails extensive manual labor from designers, involving intricate detailing and representation of complex urban elements. Therefore, accomplishing this automatically remains a longstanding challenge. Toward this problem, we propose UrbanWorld, the first generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. UrbanWorld incorporates four key stages in the generation pipeline: flexible 3D layout generation from OSM data or urban layout with semantic and height maps, urban scene design with Urban MLLM, controllable urban asset rendering via progressive 3D diffusion, and MLLM-assisted scene refinement. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves SOTA generation realism. Next, we provide qualitative results about the controllable generation capabilities of UrbanWorld using both textual and image-based prompts. Lastly, we verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments. We contribute UrbanWorld as an open-source tool available at https://github.com/Urban-World/UrbanWorld.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑