arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.00989cs.CV
收藏

POS-BERT:点云一阶段BERT预培训

POS-BERT: Point Cloud One-Stage BERT Pre-Training

Fudan University(复旦大学);Shanghai AI Lab(上海人工智能实验室)

Kexue Fu, Peng Gao, ShaoLei Liu, Renrui Zhang, Yu Qiao, Manning Wang

摘要:

最近,结合Transformer和掩码语言建模的预训练范式在NLP、图像和点云(如BERT)方面取得了巨大的成功。然而,直接将BERT从NLP扩展到点云需要在预训练之前训练固定的离散变分自编码器(dVAE),这导致称为点BERT的复杂两阶段方法。受BERT和MoCo的启发,本文提出了一种点云BERT预训练方法POS-BERT。具体来说,我们使用掩码补丁建模(MPM)任务来执行点云预训练,其目的是在相应的标记器输出的监督下恢复掩码补丁信息。与Point-BERT不同,它的标记器是经过额外训练和冻结的。我们建议使用动态更新的动量编码器作为标记器,其随着训练过程而更新并输出动态监督信号。此外,为了学习高层次的语义表示,我们结合对比学习,以最大限度地提高不同的转换点云之间的类令牌的一致性。大量的实验表明,POS-BERT可以提取高质量的预训练特征,并促进下游任务,以提高性能。POS-BERT在ModelNet 40上使用未经任何微调的预训练模型进行特征提取和线性支持向量机的训练,达到了最先进的分类精度,比Point-BERT提高了3.5%。此外,我们的方法显着改善了许多下游任务,如微调分类,Few-Shot分类,部分分割。代码和训练模型将在以下网址提供:\url{https://github.com/fukexue/POS-BERT}。

英文摘要:

Recently, the pre-training paradigm combining Transformer and masked language modeling has achieved tremendous success in NLP, images, and point clouds, such as BERT. However, directly extending BERT from NLP to point clouds requires training a fixed discrete Variational AutoEncoder (dVAE) before pre-training, which results in a complex two-stage method called Point-BERT. Inspired by BERT and MoCo, we propose POS-BERT, a one-stage BERT pre-training method for point clouds. Specifically, we use the mask patch modeling (MPM) task to perform point cloud pre-training, which aims to recover masked patches information under the supervision of the corresponding tokenizer output. Unlike Point-BERT, its tokenizer is extra-trained and frozen. We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process. Further, in order to learn high-level semantic representation, we combine contrastive learning to maximize the class token consistency between different transformation point clouds. Extensive experiments have demonstrated that POS-BERT can extract high-quality pre-training features and promote downstream tasks to improve performance. Using the pre-training model without any fine-tuning to extract features and train linear SVM on ModelNet40, POS-BERT achieves the state-of-the-art classification accuracy, which exceeds Point-BERT by 3.5\%. In addition, our approach has significantly improved many downstream tasks, such as fine-tuned classification, few-shot classification, part segmentation. The code and trained-models will be available at: \url{https://github.com/fukexue/POS-BERT}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑