采用多模态对比掩码自编码器的两阶段渐进式预训练
A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoders
AI总结：
本研究提出一种基于多模态对比掩码自编码器的两阶段渐进式RGB-D预训练方法，结合对比学习、掩码自编码、去噪与全局蒸馏，在多个数据集语义分割任务上取得优于现有方法的性能，且在小数据场景下同样有效。
摘要：
本文提出了一种用于图像理解任务的新型渐进式预训练方法，该方法利用RGB-D数据集。该方法采用多模态对比掩码自编码器（Multi-Modal Contrastive Masked Autoencoder）与去噪技术。我们提出的方法包含两个阶段：第一阶段，我们使用对比学习对模型进行预训练，以学习跨模态表征；第二阶段，我们使用掩码自编码以及扩散模型中采用的去噪/噪声预测对模型进行进一步预训练。掩码自编码侧重于利用局部空间相关性重建输入模态中缺失的图像块，而去噪则学习输入数据的高频分量。此外，第二阶段还利用第一阶段习得的知识引入了全局蒸馏。我们的方法具备可扩展性、鲁棒性，适用于RGB-D数据集的预训练。在ScanNet、NYUv2和SUN RGB-D等多个数据集上开展的大量实验证明了该方法的有效性与优越性能。具体而言，在ScanNet语义分割任务上，我们的方法相较于Mask3D的mIoU提升了+1.3%。我们还通过在语义分割任务上与当前最优方法进行对比评估，证明了该方法在小数据场景下的有效性。
英文摘要：
In this paper, we propose a new progressive pre-training method for image understanding tasks which leverages RGB-D datasets. The method utilizes Multi-Modal Contrastive Masked Autoencoder and Denoising techniques. Our proposed approach consists of two stages. In the first stage, we pre-train the model using contrastive learning to learn cross-modal representations. In the second stage, we further pre-train the model using masked autoencoding and denoising/noise prediction used in diffusion models. Masked autoencoding focuses on reconstructing the missing patches in the input modality using local spatial correlations, while denoising learns high frequency components of the input data. Moreover, it incorporates global distillation in the second stage by leveraging the knowledge acquired in stage one. Our approach is scalable, robust and suitable for pre-training RGB-D datasets. Extensive experiments on multiple datasets such as ScanNet, NYUv2 and SUN RGB-D show the efficacy and superior performance of our approach. Specifically, we show an improvement of +1.3% mIoU against Mask3D on ScanNet semantic segmentation. We further demonstrate the effectiveness of our approach in low-data regime by evaluating it for semantic segmentation task against the state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
