Terra：基于点潜在表示的可探索原生3D世界模型
Terra: Explorable Native 3D World Model with Point Latents
AI总结：
Terra提出原生3D世界模型，利用点潜在表示和稀疏点流匹配网络，实现高一致性的可探索环境生成与重建。
摘要：
世界模型因对真实世界的全面建模而受到越来越多的关注。然而，现有的大多数方法仍依赖像素对齐的表示作为世界演化的基础，忽视了物理世界固有的三维特性。这可能会损害世界模型的三维一致性并降低其建模效率。在本文中，我们提出了Terra，一种原生3D世界模型，它在内在的3D潜在空间中表示并生成可探索的环境。具体来说，我们提出了一种新颖的点到高斯变分自编码器（P2G-VAE），它将3D输入编码为潜在点表示，随后将其解码为3D高斯原语，以联合建模几何和外观。然后，我们引入了一个稀疏点流匹配网络（SPFlow）来生成潜在点表示，该网络同时对点潜在的位置和特征进行去噪。我们的Terra凭借原生3D表示和架构实现了精确的多视图一致性，并支持仅通过一次生成过程从任意视点进行灵活渲染。此外，Terra通过在点潜在空间中进行渐进式生成来实现可探索的世界建模。我们在ScanNet v2中具有挑战性的室内场景上进行了大量实验。Terra在重建和生成方面均达到了最先进的性能，并具有高3D一致性。
英文摘要：
World models have garnered increasing attention for comprehensive modeling of the real world. However, most existing methods still rely on pixel-aligned representations as the basis for world evolution, neglecting the inherent 3D nature of the physical world. This could undermine the 3D consistency and diminish the modeling efficiency of world models. In this paper, we present Terra, a native 3D world model that represents and generates explorable environments in an intrinsic 3D latent space. Specifically, we propose a novel point-to-Gaussian variational autoencoder (P2G-VAE) that encodes 3D inputs into a latent point representation, which is subsequently decoded as 3D Gaussian primitives to jointly model geometry and appearance. We then introduce a sparse point flow matching network (SPFlow) for generating the latent point representation, which simultaneously denoises the positions and features of the point latents. Our Terra enables exact multi-view consistency with native 3D representation and architecture, and supports flexible rendering from any viewpoint with only a single generation process. Furthermore, Terra achieves explorable world modeling through progressive generation in the point latent space. We conduct extensive experiments on the challenging indoor scenes from ScanNet v2. Terra achieves state-of-the-art performance in both reconstruction and generation with high 3D consistency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
