GS-CLIP：用于基于真实世界数据的对比语言-图像-3D预训练的Gaussian Splatting
GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data
AI总结：
提出GS-CLIP框架，首次将3DGS引入多模态预训练，通过3D Encoder和Gaussian-Aware Fusion对齐3D与语言图像空间，显著提升3D表示性能。
摘要：
表示为点云的3D形状在多模态预训练中取得了进展，以对齐图像和语言描述，这对于物体识别、分类和检索至关重要。然而，点云的离散表示丢失了物体的表面形状信息，并在渲染结果与2D对应关系之间产生了差距。为了解决这个问题，我们提出了GS-CLIP，首次尝试将3DGS（3D Gaussian Splatting）引入多模态预训练以增强3D表示。GS-CLIP利用预训练的视觉-语言模型，在大量真实世界图像-文本对上学习共同的视觉和文本空间，然后学习一个3D Encoder来对齐每个物体优化的3DGS。此外，提出了一种新颖的Gaussian-Aware Fusion来提取和融合全局显式特征。作为语言-图像-3D预训练的通用框架，GS-CLIP与3D骨干网络无关。在具有挑战性的实验上表明，GS-CLIP显著改进了现有技术，超越了先前的最佳结果。
英文摘要：
3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions, which is curial to object identification, classification, and retrieval. However, the discrete representations of point cloud lost the object's surface shape information and creates a gap between rendering results and 2D correspondences. To address this problem, we propose GS-CLIP for the first attempt to introduce 3DGS (3D Gaussian Splatting) into multimodal pre-training to enhance 3D representation. GS-CLIP leverages a pre-trained vision-language model for a learned common visual and textual space on massive real world image-text pairs and then learns a 3D Encoder for aligning 3DGS optimized per object. Additionally, a novel Gaussian-Aware Fusion is proposed to extract and fuse global explicit feature. As a general framework for language-image-3D pre-training, GS-CLIP is agnostic to 3D backbone networks. Experiments on challenging shows that GS-CLIP significantly improves the state-of-the-art, outperforming the previously best results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
