在对比语言-图像-3D预训练中塑造整体3D表示
Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training
AI总结：
本文提出MixCon3D方法，通过融合点云与多视图渲染图像构建整体3D表示并进行语言-3D对比学习，在Objaverse-LVIS等基准上超越先前最优性能5.7%，并展示了在检索和字幕任务中的通用性。
摘要：
对比学习已成为3D开放世界理解的一种有前景的范式，即分别将点云表示对齐到图像和文本嵌入空间。本文提出MixCon3D，一种简单而有效的方法，旨在对比语言-图像-3D预训练中塑造整体3D表示。与仅使用点云不同，我们从互补视角构建3D对象级表示，例如多视图渲染图像与点云。然后，MixCon3D执行语言-3D对比学习，全面刻画真实世界3D对象并增强文本对齐。此外，我们率先对3D对比学习范式的各种训练配方进行了首次全面研究，构建了具有改进性能的坚实基线。在三个代表性基准上开展的大量实验表明，我们的方法显著优于基线，在具有挑战性的1,156类Objaverse-LVIS数据集上超过先前最先进性能5.7%。MixCon3D的多功能性在文本到3D检索和点云字幕等应用中得到了展示，进一步证明了其在多样场景中的有效性。代码可在https://github.com/UCSC-VLAA/MixCon3D获取。
英文摘要：
Contrastive learning has emerged as a promising paradigm for 3D open-world understanding, i.e., aligning point cloud representation to image and text embedding space individually. In this paper, we introduce MixCon3D, a simple yet effective method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training. In contrast to point cloud only, we develop the 3D object-level representation from complementary perspectives, e.g., multi-view rendered images with the point cloud. Then, MixCon3D performs language-3D contrastive learning, comprehensively depicting real-world 3D objects and bolstering text alignment. Additionally, we pioneer the first thorough investigation of various training recipes for the 3D contrastive learning paradigm, building a solid baseline with improved performance. Extensive experiments conducted on three representative benchmarks reveal that our method significantly improves over the baseline, surpassing the previous state-of-the-art performance on the challenging 1,156-category Objaverse-LVIS dataset by 5.7%. The versatility of MixCon3D is showcased in applications such as text-to-3D retrieval and point cloud captioning, further evidencing its efficacy in diverse scenarios. The code is available at https://github.com/UCSC-VLAA/MixCon3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
