任何3D场景都值得1000个标记：面向大规模场景生成的3D grounded表示
Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
AI总结：
本文提出在隐式3D潜在空间中直接生成3D场景，解决传统2D方法在3D空间扩展中的不足，通过3DRAE和3DDiT实现高效且一致的3D生成。
摘要：
3D场景生成长期以来由2D多视图或视频扩散模型主导。这不仅由于缺乏场景级3D潜在表示，还因为大多数场景级3D视觉数据以多视图图像或视频形式存在，自然兼容2D扩散架构。通常，这些2D方法将3D空间扩展降级为2D时间扩展，导致两个根本问题：（i）通过2D视图表示3D场景导致显著的表示冗余；（ii）基于2D的潜在空间限制了生成3D场景的空间一致性。本文首次提出在隐式3D潜在空间中直接进行3D场景生成，以解决这些限制。首先，我们重新利用冻结的2D表示编码器构建我们的3D表示自编码器（3DRAE），将视图耦合的2D语义表示转换为视图解耦的3D潜在表示。这使我们能够用固定复杂度和丰富语义表示从任意数量的视图中观察到的3D场景，无论分辨率和宽高比如何。然后我们引入3D扩散变换器（3DDiT），在该3D潜在空间中进行扩散建模，实现高效且一致的3D场景生成，同时支持多种条件配置。此外，由于我们的方法直接生成3D场景表示，可以沿任意相机轨迹解码为图像和可选点图，而无需每个轨迹的扩散采样过程，这在2D方法中很常见。
英文摘要：
3D scene generation has long been dominated by 2D multi-view or video diffusion models. This is due not only to the lack of scene-level 3D latent representation, but also to the fact that most scene-level 3D visual data exists in the form of multi-view images or videos, which are naturally compatible with 2D diffusion architectures. Typically, these 2D-based approaches degrade 3D spatial extrapolation to 2D temporal extension, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated 3D scenes. In this paper, we propose, for the first time, to perform 3D scene generation directly within an implicit 3D latent space to address these limitations. First, we repurpose frozen 2D representation encoders to construct our 3D Representation Autoencoder (3DRAE), which grounds view-coupled 2D semantic representations into a view-decoupled 3D latent representation. This enables representing 3D scenes observed from arbitrary numbers of views--at any resolution and aspect ratio--with fixed complexity and rich semantics. Then we introduce 3D Diffusion Transformer (3DDiT), which performs diffusion modeling in this 3D latent space, achieving remarkably efficient and spatially consistent 3D scene generation while supporting diverse conditioning configurations. Moreover, since our approach directly generates a 3D scene representation, it can be decoded to images and optional point maps along arbitrary camera trajectories without requiring per-trajectory diffusion sampling pass, which is common in 2D-based approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
