OneWorld: 通过3D统一表示自编码器驯服场景生成
OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder
AI总结:
OneWorld通过3D统一表示自编码器直接在3D空间中进行扩散,解决跨视角一致性和几何一致性问题,实验表明其生成的3D场景质量优于现有2D方法。
摘要:
现有基于扩散的3D场景生成方法主要在2D图像/视频潜在空间中操作,导致维持跨视角外观和几何一致性具有内在挑战。为弥合这一差距,我们提出了OneWorld框架,该框架在连贯的3D表示空间中直接进行扩散。核心方法是3D统一表示自编码器(3D-URAE),它利用预训练的3D基础模型,并通过注入外观和蒸馏语义到统一的3D潜在空间来增强其几何中心性质。此外,我们引入了token级跨视角对应(CVC)一致性损失,以显式地强制跨视角的结构对齐,并提出了流形漂移强制(MDF)以缓解训练-推理暴露偏差,并通过混合漂移和原始表示来塑造稳健的3D流形。全面的实验表明,OneWorld生成的高质量3D场景在跨视角一致性方面优于最先进的2D方法。我们的代码将在https://github.com/SensenGao/OneWorld上提供。
英文摘要:
Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。