UniGS：基于高斯溅射的统一语言-图像-三维预训练
UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting
AI总结：
UniGS将三维高斯溅射融入多模态预训练，通过高斯感知引导模块对齐语言-图像-三维表示，在零样本分类、检索和开放世界理解任务上显著超越现有最优方法。
摘要：
近年来，多模态三维预训练方法的进展在文本、图像和点云的联合表示学习方面展现出了良好的效果。然而，采用点云作为三维表示无法完全捕捉三维世界的复杂性，并且在离散点与图像的密集二维像素之间表现出明显的差距。为解决这一问题，我们提出了UniGS，将三维高斯溅射（3DGS）集成到多模态预训练中，以增强三维表示。我们首先依靠3DGS表示将三维世界建模为带有颜色和不透明度的三维高斯集合，在包含三维场景所有信息的同时，与二维图像建立强有力的联系。然后，为实现语言-图像-三维预训练，UniGS从预训练的视觉-语言模型开始，通过大量真实世界的图像-文本对建立共享的视觉和文本空间。随后，UniGS采用三维编码器将优化的3DGS与语言-图像表示对齐，以学习统一的多模态表示。为了促进三维编码器提取全局显式三维特征并实现更好的跨模态对齐，我们还引入了一个新颖的高斯感知引导模块，该模块引导三维领域细粒度表示的学习。通过在Objaverse、ABO、MVImgNet和SUN RGBD数据集上进行的广泛实验，包括零样本分类、文本驱动的检索和开放世界理解任务，我们证明了UniGS在学习更通用且更强对齐的多模态表示方面的有效性。具体而言，UniGS在不同三维任务中取得了领先的结果，相比之前的SOTA方法Uni3D有显著提升，包括零样本分类（+9.36%）、文本驱动的检索（+4.3%）和开放世界理解（+7.92%）。
英文摘要：
Recent advancements in multi-modal 3D pre-training methods have shown promising efficacy in learning joint representations of text, images, and point clouds. However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits a noticeable gap between the discrete points and the dense 2D pixels of images. To tackle this issue, we propose UniGS, integrating 3D Gaussian Splatting (3DGS) into multi-modal pre-training to enhance the 3D representation. We first rely on the 3DGS representation to model the 3D world as a collection of 3D Gaussians with color and opacity, incorporating all the information of the 3D scene while establishing a strong connection with 2D images. Then, to achieve Language-Image-3D pertaining, UniGS starts with a pre-trained vision-language model to establish a shared visual and textual space through extensive real-world image-text pairs. Subsequently, UniGS employs a 3D encoder to align the optimized 3DGS with the Language-Image representations to learn unified multi-modal representations. To facilitate the extraction of global explicit 3D features by the 3D encoder and achieve better cross-modal alignment, we additionally introduce a novel Gaussian-Aware Guidance module that guides the learning of fine-grained representations of the 3D domain. Through extensive experiments across the Objaverse, ABO, MVImgNet and SUN RGBD datasets with zero-shot classification, text-driven retrieval and open-world understanding tasks, we demonstrate the effectiveness of UniGS in learning a more general and stronger aligned multi-modal representation. Specifically, UniGS achieves leading results across different 3D tasks with remarkable improvements over previous SOTA, Uni3D, including on zero-shot classification (+9.36%), text-driven retrieval (+4.3%) and open-world understanding (+7.92%).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
