PanopticSplatting：端到端全景高斯溅射
PanopticSplatting: End-to-End Panoptic Gaussian Splatting
AI总结：
针对现有多阶段3D全景重建方法误差累积、依赖手工组件的问题，提出端到端系统PanopticSplatting，通过查询引导局部分割、标签融合与扭曲等技术，在多数据集上取得优于现有方法的性能，且具备良好的泛化性与鲁棒性。
摘要：
开放词汇全景重建是同时进行场景重建与理解的一项挑战性任务。近期已有研究提出基于高斯溅射的3D场景理解方法，但这类方法为多阶段流程，存在误差累积以及依赖手工设计组件的问题。为简化流程并实现全局优化，我们提出PanopticSplatting，一种面向开放词汇全景重建的端到端系统。我们的方法引入了带局部交叉注意力的查询引导高斯分割，无需跨帧关联即可以端到端方式提升2D实例掩码的维度。视锥体内的局部交叉注意力有效降低了训练内存开销，使模型更适用于包含更多高斯体和物体的大场景。此外，为解决2D伪掩码中的标签噪声问题，我们提出标签融合方法以提升3D分割的一致性、减少噪声漂浮物，同时对2D预测结果进行标签扭曲，增强多视图一致性与分割精度。在ScanNet-V2和ScanNet++数据集上的3D场景全景重建任务中，我们的方法相较于基于NeRF和基于高斯的全景重建方法均展现出优异性能。此外，PanopticSplatting可轻松推广至多种高斯溅射变体，我们也在不同的高斯基础模型上验证了其鲁棒性。
英文摘要：
Open-vocabulary panoptic reconstruction is a challenging task for simultaneous scene reconstruction and understanding. Recently, methods have been proposed for 3D scene understanding based on Gaussian splatting. However, these methods are multi-staged, suffering from the accumulated errors and the dependence of hand-designed components. To streamline the pipeline and achieve global optimization, we propose PanopticSplatting, an end-to-end system for open-vocabulary panoptic reconstruction. Our method introduces query-guided Gaussian segmentation with local cross attention, lifting 2D instance masks without cross-frame association in an end-to-end way. The local cross attention within view frustum effectively reduces the training memory, making our model more accessible to large scenes with more Gaussians and objects. In addition, to address the challenge of noisy labels in 2D pseudo masks, we propose label blending to promote consistent 3D segmentation with less noisy floaters, as well as label warping on 2D predictions which enhances multi-view coherence and segmentation accuracy. Our method demonstrates strong performances in 3D scene panoptic reconstruction on the ScanNet-V2 and ScanNet++ datasets, compared with both NeRF-based and Gaussian-based panoptic reconstruction methods. Moreover, PanopticSplatting can be easily generalized to numerous variants of Gaussian splatting, and we demonstrate its robustness on different Gaussian base models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
