GGRt：迈向无位姿、可泛化的实时 3D Gaussian Splatting
GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time
AI总结：
提出无位姿、可泛化的实时 3D-GS 框架 GGRt，通过 IPO-Net 与 G-3DG 联合学习、延迟反向传播和渐进式高斯缓存，实现高分辨率实时渲染并达到 SOTA。
摘要：
本文提出 GGRt，这是一种用于可泛化新视角合成的新颖方法，可消除对真实相机位姿的需求、处理高分辨率图像的复杂性以及冗长的优化过程，从而增强 3D Gaussian Splatting（3D-GS）在真实场景中的适用性。具体而言，我们设计了一个新颖的联合学习框架，由 Iterative Pose Optimization Network（IPO-Net，迭代位姿优化网络）和 Generalizable 3D-Gaussians（G-3DG，可泛化三维高斯）模型组成。借助联合学习机制，所提出的框架能够从图像观测中内在地估计鲁棒的相对位姿信息，从而从根本上降低对真实相机位姿的要求。此外，我们实现了一种延迟反向传播机制，支持高分辨率训练和推理，克服了以往方法的分辨率限制。为提升速度和效率，我们进一步引入渐进式高斯缓存模块，该模块在训练和推理期间动态调整。作为首个无位姿的可泛化 3D-GS 框架，GGRt 实现了不低于 5 FPS 的推理速度以及不低于 100 FPS 的实时渲染。通过大量实验，我们证明该方法在推理速度和效果上优于现有基于 NeRF 的无位姿技术，并且能够接近基于真实位姿的 3D-GS 方法。我们的贡献推动计算机视觉与计算机图形学在实际应用中的融合取得重大飞跃，在 LLFF、KITTI 和 Waymo Open 数据集上提供了最先进结果，并支持面向沉浸式体验的实时渲染。
英文摘要：
This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at $\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
