CLAP：通过曲率采样和原型学习实现融合3D感知的无监督3D表示学习
CLAP: Unsupervised 3D Representation Learning for Fusion 3D Perception via Curvature Sampling and Prototype Learning
AI总结：
CLAP通过曲率采样和原型学习，在无监督3D表示学习中实现图像与点云的联合预训练，提升融合3D感知的性能。
摘要：
无监督3D表示学习减少了为融合感知任务标注多模态3D数据的负担。在不同的预训练范式中，基于可微渲染的方法显示出最大的潜力。然而，由于处理大点云与图像的计算成本，现有工作分别对每种模态进行预训练。因此，图像的高层语义和点云的3D结构之间的相互利益尚未被利用。为了解决这一差距，我们提出了一种联合无监督可微渲染预训练方法，用于图像和点云，称为CLAP，即曲率采样和可学习原型。具体而言，我们的方法通过曲率采样来选择更有信息量的点/像素进行预训练，以克服计算障碍。为了揭示互补性带来的性能收益，我们提出使用可学习原型在共同的特征空间中表示3D场景的部分，并采用期望最大化训练方案将每种模态的嵌入关联到原型。我们进一步提出一种交换预测损失，通过原型探索它们的相互作用，并结合Gram矩阵正则化项以维持训练稳定性。在NuScenes和Waymo数据集上的实验表明，CLAP相比先前的SOTA预训练方法实现了高达100%的性能提升。
英文摘要：
Unsupervised 3D representation learning reduces the burden of labeling multimodal 3D data for fusion perception tasks. Among different pre-training paradigms, differentiable-rendering-based methods have shown most promise. However, existing works separately conduct pre-training for each modalities due to computational costs of processing large point clouds with images. As such, mutual benefit of high-level semantics (from image) and 3D structure (from point cloud) has not been exploited. To address this gap, we propose a joint unsupervised differentiable-rendering-based pre-training method for images and point clouds, termed CLAP, short for Curvature sampLing and leArnable Prototype. Specifically, our method overcomes the computational hurdle by Curvature Sampling to select the more informative points/pixels for pre-training. To uncover the performance benefits brought by their complementarity, we propose to use learnable prototypes to represent parts of the 3D scenes in a common feature space and an Expectation-Maximization training scheme to associate embeddings of each modality to prototypes. We further propose a swapping prediction loss that explores their interplay through prototypes along with a Gram Matrix Regularization term to maintain training stability. Experiments on NuScenes and Waymo datasets show that CLAP achieves up to 100% more performance gain as compared to previous SOTA pre-training methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
