Triplane 遇上高斯泼溅：基于Transformer的快速可泛化单视图3D重建
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers
AI总结：
针对现有单视图3D重建方法优化渲染慢的问题，提出结合点解码器与triplane解码器的Transformer架构，用混合Triplane-Gaussian表示实现前馈式快速高质量重建，在合成与真实数据上均优于现有SOTA方法。
摘要：
生成模型的发展推动了单图像3D重建领域的最新进展，其中基于分数蒸馏采样（Score Distillation Sampling，SDS）以及扩散模型在3D领域适配的方法尤为突出。尽管这些方法取得了进展，但它们往往因优化或渲染过程缓慢而存在局限，导致训练和优化时间过长。本文提出了一种新颖的单视图重建方法，可通过前馈推理从单张图像高效生成3D模型。我们的方法采用两个基于Transformer的网络，即点解码器和triplane解码器，通过混合Triplane-Gaussian中间表示来重建3D物体。这种混合表示实现了平衡：相比隐式表示拥有更快的渲染速度，同时比显式表示拥有更优的渲染质量。点解码器用于从单张图像生成点云，提供显式表示，随后triplane解码器利用该表示为每个点查询高斯特征。这一设计解决了直接回归显式3D高斯属性时因属性无结构特性带来的挑战。之后，3D高斯由多层感知器（MLP）解码，以通过泼溅（splatting）实现快速渲染。两个解码器均基于可扩展的Transformer架构构建，并已在大规模3D数据集上完成高效训练。在合成数据集和真实世界图像上开展的评估表明，与此前的最优技术相比，我们的方法不仅实现了更高的质量，还保证了更快的运行速度。项目页面请访问https://zouzx.github.io/TriplaneGaussian/。
英文摘要：
Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at https://zouzx.github.io/TriplaneGaussian/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
