Multi-CLIP:面向3D场景问答任务的对比式视觉语言预训练
Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
AI总结：
针对2D视觉语言知识迁移至3D场景任务的研究不足问题，提出3D预训练方法Multi-CLIP，通过对比学习对齐3D点云特征与CLIP空间的2D多视图、文本特征，在3D-VQA和3D-SQA任务上超越现有最优水平。
摘要：
训练模型将2D图像中的常识语言知识与视觉概念应用于3D场景理解，是研究人员近期才开始探索的一个极具前景的方向。然而，2D蒸馏得到的知识能否为3D视觉问答等下游3D视觉语言任务提供有效的表征，目前仍未得到充分研究。本文提出了一种新型3D视觉语言预训练方法Multi-CLIP，该方法可使模型学习到具备语言基础、可迁移的3D场景点云表征。我们通过对比目标最大化编码后的3D场景特征与CLIP空间中对应的2D多视图图像、文本嵌入之间的一致性，以此利用CLIP模型的表征能力。为验证所提方法，我们选取了具有挑战性的下游任务：3D视觉问答（3D-VQA）和3D情境问答（3D-SQA）。为此，我们设计了基于多模态transformer的新型架构，并证明了该预训练方法能够提升其性能。定量与定性实验结果表明，Multi-CLIP在3D-VQA和3D-SQA下游任务上的表现均优于现有最优方法，且能构建出结构良好的3D场景特征空间。
英文摘要：
Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. However, it still remains understudied whether 2D distilled knowledge can provide useful representations for downstream 3D vision-language tasks such as 3D question answering. In this paper, we propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations. We leverage the representational power of the CLIP model by maximizing the agreement between the encoded 3D scene features and the corresponding 2D multi-view image and text embeddings in the CLIP space via a contrastive objective. To validate our approach, we consider the challenging downstream tasks of 3D Visual Question Answering (3D-VQA) and 3D Situated Question Answering (3D-SQA). To this end, we develop novel multi-modal transformer-based architectures and we demonstrate how our pre-training method can benefit their performance. Quantitative and qualitative experimental results show that Multi-CLIP outperforms state-of-the-art works across the downstream tasks of 3D-VQA and 3D-SQA and leads to a well-structured 3D scene feature space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
