arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.04098cs.CV

EPCL：冻结的 CLIP Transformer 是一种高效的点云编码器

EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder

Shanghai AI Lab（上海人工智能实验室）；Jiangxi University of Finance and Economics（江西财经大学）

Xiaoshui Huang, Zhou Huang, Sheng Li, Wentao Qu, Tong He, Yuenan Hou, Yifan Zuo, Wanli Ouyang

备注：
AAAI2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 EPCL，通过点云 tokenizer 将点云映射到类图像 token 空间，并借助冻结 CLIP transformer 在无配对 2D-3D 数据下学习表示，在多项 3D 任务和基准上取得显著提升。

摘要：

预训练—微调范式凭借其预训练模型的高质量表示能力和可迁移性，已在 NLP 和 2D 图像领域取得巨大成功。然而，由于点云序列数量有限，在 3D 点云领域预训练这样的强模型十分困难。本文提出 Efficient Point Cloud Learning（EPCL，高效点云学习），这是一种有效且高效的点云学习器，可利用冻结的 CLIP transformer 直接训练高质量点云模型。我们的 EPCL 通过在语义上对齐图像特征和点云特征来连接 2D 与 3D 模态，且不需要成对的 2D-3D 数据。具体而言，输入点云被划分为一系列局部块，并由所设计的点云 tokenizer 转换为 token 嵌入。这些 token 嵌入与一个任务 token 拼接后，输入冻结的 CLIP transformer 以学习点云表示。其直觉在于，所提出的点云 tokenizer 将输入点云投影到一个与 2D 图像相似的统一 token 空间中。在 3D 检测、语义分割、分类和少样本学习上的综合实验表明，CLIP transformer 可以作为高效的点云编码器，并且我们的方法在室内和室外基准上均取得了有前景的性能。特别是，与同期预训练模型相比，我们的 EPCL 在 ScanNet V2 检测上带来 19.7 AP50 的性能提升，在 S3DIS 分割上带来 4.4 mIoU 的提升，并在 SemanticKITTI 分割上带来 1.2 mIoU 的提升。代码位于 https://github.com/XiaoshuiHuang/EPCL。

英文摘要：

The pretrain-finetune paradigm has achieved great success in NLP and 2D image fields because of the high-quality representation ability and transferability of their pretrained models. However, pretraining such a strong model is difficult in the 3D point cloud field due to the limited amount of point cloud sequences. This paper introduces \textbf{E}fficient \textbf{P}oint \textbf{C}loud \textbf{L}earning (EPCL), an effective and efficient point cloud learner for directly training high-quality point cloud models with a frozen CLIP transformer. Our EPCL connects the 2D and 3D modalities by semantically aligning the image features and point cloud features without paired 2D-3D data. Specifically, the input point cloud is divided into a series of local patches, which are converted to token embeddings by the designed point cloud tokenizer. These token embeddings are concatenated with a task token and fed into the frozen CLIP transformer to learn point cloud representation. The intuition is that the proposed point cloud tokenizer projects the input point cloud into a unified token space that is similar to the 2D images. Comprehensive experiments on 3D detection, semantic segmentation, classification and few-shot learning demonstrate that the CLIP transformer can serve as an efficient point cloud encoder and our method achieves promising performance on both indoor and outdoor benchmarks. In particular, performance gains brought by our EPCL are $\textbf{19.7}$ AP$_{50}$ on ScanNet V2 detection, $\textbf{4.4}$ mIoU on S3DIS segmentation and $\textbf{1.2}$ mIoU on SemanticKITTI segmentation compared to contemporary pretrained models. Code is available at \url{https://github.com/XiaoshuiHuang/EPCL}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑