arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09394cs.CVcs.GR
收藏

WonderWorld:从单张图像进行交互式3D场景生成

WonderWorld: Interactive 3D Scene Generation from a Single Image

Stanford University(斯坦福大学);MIT(麻省理工学院)

Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T. Freeman, Jiajun Wu

AI总结:

WonderWorld提出快速分层高斯曲面片表示和引导深度扩散,从单张图像在10秒内生成连贯3D场景,支持实时交互。

摘要:

我们提出了WonderWorld,一个用于交互式3D场景生成的新框架,使用户能够交互式地指定场景内容和布局,并以低延迟看到生成的场景。主要挑战在于实现3D场景的快速生成。现有的场景生成方法在速度上有所不足,因为它们通常需要(1)逐步生成多个视图和深度图,以及(2)对场景几何表示进行耗时的优化。我们引入了快速分层高斯曲面片(FLAGS)作为我们的场景表示,并提出了从单一视图生成它的算法。我们的方法不需要多个视图,并利用基于几何的初始化,显著减少了优化时间。另一个挑战是生成连贯的几何结构,使所有场景能够连接起来。我们引入了引导深度扩散,允许对深度估计进行部分条件化。WonderWorld在单个A6000 GPU上不到10秒内生成连贯且多样化的3D场景,实现实时用户交互和探索。我们展示了WonderWorld在虚拟环境中用户驱动内容创建和探索的潜力。我们发布完整的代码和软件以确保可复现性。项目网站:https://kovenyu.com/WonderWorld/。

英文摘要:

We present WonderWorld, a novel framework for interactive 3D scene generation that enables users to interactively specify scene contents and layout and see the created scenes in low latency. The major challenge lies in achieving fast generation of 3D scenes. Existing scene generation approaches fall short of speed as they often require (1) progressively generating many views and depth maps, and (2) time-consuming optimization of the scene geometry representations. We introduce the Fast Layered Gaussian Surfels (FLAGS) as our scene representation and an algorithm to generate it from a single view. Our approach does not need multiple views, and it leverages a geometry-based initialization that significantly reduces optimization time. Another challenge is generating coherent geometry that allows all scenes to be connected. We introduce the guided depth diffusion that allows partial conditioning of depth estimation. WonderWorld generates connected and diverse 3D scenes in less than 10 seconds on a single A6000 GPU, enabling real-time user interaction and exploration. We demonstrate the potential of WonderWorld for user-driven content creation and exploration in virtual environments. We release full code and software for reproducibility. Project website: https://kovenyu.com/WonderWorld/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑