arXiv 2604.165522026-04-29cs.CVcs.AI

通过自回归3D扩散模型生成布局和形状

Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion

Meta Reality Labs（Meta现实实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Texas at Austin（德克萨斯大学奥斯汀分校）

Zhenggang Tang, Yuehao Wang, Yuchen Fan, Jun-Kun Chen, Yu-Ying Yeh, Kihyuk Sohn, Zhangyang Wang, Qixing Huang, Alexander Schwing, Rakesh Ranjan, Dilin Wang, Zhicheng Yan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种新的文本到场景生成方法，通过自回归3D扩散模型生成布局和形状，解决文本描述与生成场景不一致的问题。

摘要：

近期的文本到场景生成方法大幅减少了创建3D场景所需的手动工作量。然而，这些方法要么专注于生成场景布局，要么专注于生成物体，很少同时生成两者。生成的场景布局即使在使用LLM帮助下也往往很简单。此外，生成的场景通常与包含非平凡物体形状、外观和空间排列描述的文本输入不一致。我们提出了一种新的顺序文本到场景生成范式，并提出了一种新的生成模型用于交互式场景创建。核心是3D自回归扩散模型3D-ARD+，它统一了对多模态令牌序列的自回归生成和下一步物体3D潜在的扩散生成。为了生成下一个物体，模型使用一个自回归步骤在场景空间中生成粗粒度的3D潜在，并根据当前已见的文本指令和已合成的3D场景进行条件生成。然后使用第二步生成在较小物体空间中的3D潜在，这些潜在可以解码为精细的物体几何和外观。我们整理了一个包含230万室内场景及其配对文本指令的数据集进行训练。我们在具有挑战性的场景上评估了7B的3D-ARD+，并展示了该模型能够根据文本指令生成并放置符合非平凡空间布局和语义的物体。

英文摘要：

Recent text-to-scene generation approaches largely reduced the manual efforts required to create 3D scenes. However, their focus is either to generate a scene layout or to generate objects, and few generate both. The generated scene layout is often simple even with LLM's help. Moreover, the generated scene is often inconsistent with the text input that contains non-trivial descriptions of the shape, appearance, and spatial arrangement of the objects. We present a new paradigm of sequential text-to-scene generation and propose a novel generative model for interactive scene creation. At the core is a 3D Autoregressive Diffusion model 3D-ARD+, which unifies the autoregressive generation over a multimodal token sequence and diffusion generation of next-object 3D latents. To generate the next object, the model uses one autoregressive step to generate the coarse-grained 3D latents in the scene space, conditioned on both the current seen text instructions and already synthesized 3D scene. It then uses a second step to generate the 3D latents in the smaller object space, which can be decoded into fine-grained object geometry and appearance. We curate a large dataset of 230K indoor scenes with paired text instructions for training. We evaluate 7B 3D-ARD+, on challenging scenes, and showcase the model can generate and place objects following non-trivial spatial layout and semantics prescribed by the text instructions.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。