arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2412.19142cs.CV

CLIP-GS: 通过3D高斯点划法统一视觉-语言表示

CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splatting

Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）；National University of Singapore（新加坡国立大学）；Meituan（美团）；Georgia Institute of Technology（佐治亚理工学院）；Picsart AI Research (PAIR)（Picsart AI研究（PAIR））

Siyu Jiao, Haoye Dong, Yuyang Yin, Zequn Jie, Yinlong Qian, Yao Zhao, Humphrey Shi, Yunchao Wei

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

CLIP-GS通过3D高斯点划法统一视觉-语言表示，利用对比损失和图像投票损失提升多模态检索和分类性能。

摘要：

近期3D多模态学习的研究取得了显著进展。然而，通常3D多模态模型只能处理点云。与新兴的3D表示技术3D高斯点划法（3DGS）相比，空间稀疏的点云无法描绘3D物体的纹理信息，导致重建能力较差。这一限制制约了基于点云的3D多模态表示学习的潜力。在本文中，我们提出了CLIP-GS，一种基于3DGS的新型多模态表示学习框架。我们引入了GS Tokenizer生成序列化的高斯令牌，然后通过预初始化权重来自点云模型的transformer层进行处理，得到3DGS嵌入。CLIP-GS利用3DGS与CLIP的视觉-文本嵌入之间的对比损失，并引入图像投票损失来指导梯度优化的方向性和收敛性。此外，我们开发了一种高效生成3DGS、图像和文本三元组的方法，使CLIP-GS能够学习统一的多模态表示。利用高度对齐的多模态表示，CLIP-GS展示了其通用性和在各种3D任务上的优越表现，包括多模态检索、零样本和少样本分类。

英文摘要：

Recent works in 3D multimodal learning have made remarkable progress. However, typically 3D multimodal models are only capable of handling point clouds. Compared to the emerging 3D representation technique, 3D Gaussian Splatting (3DGS), the spatially sparse point cloud cannot depict the texture information of 3D objects, resulting in inferior reconstruction capabilities. This limitation constrains the potential of point cloud-based 3D multimodal representation learning. In this paper, we present CLIP-GS, a novel multimodal representation learning framework grounded in 3DGS. We introduce the GS Tokenizer to generate serialized gaussian tokens, which are then processed through transformer layers pre-initialized with weights from point cloud models, resulting in the 3DGS embeddings. CLIP-GS leverages contrastive loss between 3DGS and the visual-text embeddings of CLIP, and we introduce an image voting loss to guide the directionality and convergence of gradient optimization. Furthermore, we develop an efficient way to generate triplets of 3DGS, images, and text, facilitating CLIP-GS in learning unified multimodal representations. Leveraging the well-aligned multimodal representations, CLIP-GS demonstrates versatility and outperforms point cloud-based models on various 3D tasks, including multimodal retrieval, zero-shot, and few-shot classification.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑