CLIP引导的视觉-语言预训练用于3D场景问答
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
AI总结：
本文提出CLIP引导的3D视觉-语言预训练方法，通过将3D点云特征与CLIP的2D图像和文本嵌入对齐，学习可迁移的3D场景表示，并在3D视觉问答任务上取得优于现有方法的性能。
摘要：
训练模型将2D图像中的语言知识和视觉概念应用于3D世界理解，是一个研究人员最近才开始探索的有前景的方向。在这项工作中，我们设计了一种新颖的3D预训练视觉-语言方法，帮助模型学习具有语义意义且可迁移的3D场景点云表示。我们通过将编码后的3D场景特征与CLIP生成的相应2D图像和文本嵌入对齐，将流行的CLIP模型的表示能力注入到我们的3D编码器中。为了评估我们模型的3D世界推理能力，我们在3D视觉问答的下游任务上对其进行评估。实验的定量和定性结果表明，我们的预训练方法在该任务上优于最先进的工作，并产生了可解释的3D场景特征表示。
英文摘要：
Training models to apply linguistic knowledge and visual concepts from 2D images to 3D world understanding is a promising direction that researchers have only recently started to explore. In this work, we design a novel 3D pre-training Vision-Language method that helps a model learn semantically meaningful and transferable 3D scene point cloud representations. We inject the representational power of the popular CLIP model into our 3D encoder by aligning the encoded 3D scene features with the corresponding 2D image and text embeddings produced by CLIP. To assess our model's 3D world reasoning capability, we evaluate it on the downstream task of 3D Visual Question Answering. Experimental quantitative and qualitative results show that our pre-training method outperforms state-of-the-art works in this task and leads to an interpretable representation of 3D scene features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
