arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.10007cs.CV
收藏

P3P:用于扩展基于3D体素的掩码自编码器的伪3D预训练

P3P: Pseudo-3D Pre-training for Scaling 3D Voxel-based Masked Autoencoders

SIGS, Tsinghua University(清华大学信号与信息处理系);Youtu Lab, Tencent(腾讯优图实验室);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Xuechao Chen, Ying Chen, Jialin Li, Qiang Nie, Hanqiu Deng, Yong Liu, Qixing Huang, Yang Li

AI总结:

针对3D预训练数据扩展难题,提出P3P伪3D预训练框架,借助深度估计模型将图像转为3D预训练数据,搭配线性时间复杂度分词器与新重建目标,在多项3D任务上达SOTA。

摘要:

3D预训练对3D感知任务至关重要。然而,受限于收集干净完整的3D数据的难度,3D预训练一直面临数据扩展的挑战。在本研究中,我们提出了一种新型自监督预训练框架,该框架利用大型深度估计模型将数百万张图像纳入3D预训练语料库。新的预训练语料库给模型的表征能力和嵌入效率带来了新的挑战。以往的预训练方法依赖最远点采样和k近邻算法来嵌入固定数量的3D token。然而,当需要嵌入数百万个样本(点数量从1000到100000不等,范围广泛)时,这些方法显得力不从心。相比之下,我们提出了一种具有线性时间复杂度的分词器,能够高效嵌入数量灵活的token。相应地,我们提出了一种新的3D重建目标,以配合我们的3D分词器使用。我们的方法在3D分类、小样本学习和3D分割任务上均取得了最先进的性能。代码可在https://github.com/XuechaoChen/P3P-MAE获取。

英文摘要:

3D pre-training is crucial to 3D perception tasks. Nevertheless, limited by the difficulties in collecting clean and complete 3D data, 3D pre-training has persistently faced data scaling challenges. In this work, we introduce a novel self-supervised pre-training framework that incorporates millions of images into 3D pre-training corpora by leveraging a large depth estimation model. New pre-training corpora encounter new challenges in representation ability and embedding efficiency of models. Previous pre-training methods rely on farthest point sampling and k-nearest neighbors to embed a fixed number of 3D tokens. However, these approaches prove inadequate when it comes to embedding millions of samples that feature a diverse range of point numbers, spanning from 1,000 to 100,000. In contrast, we propose a tokenizer with linear-time complexity, which enables the efficient embedding of a flexible number of tokens. Accordingly, a new 3D reconstruction target is proposed to cooperate with our 3D tokenizer. Our method achieves state-of-the-art performance in 3D classification, few-shot learning, and 3D segmentation. Code is available at https://github.com/XuechaoChen/P3P-MAE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑