通过3D到多视图掩码学习器进行点云自监督学习
Point Cloud Self-supervised Learning via 3D to Multi-view Masked Learner
AI总结：
针对多模态MAE依赖外部2D输入并削弱3D几何学习的问题，提出仅用3D点云生成多视图特征的Multi-View ML，结合自编码器、MSMH注意力和两阶段自训练，在多项3D任务上取得领先性能。
摘要：
近年来，多模态掩码自编码器（MAE）被引入3D自监督学习，通过利用2D和3D数据来捕获更丰富的跨模态表示，从而提供增强的特征学习。然而，这些方法存在两个局限：（1）它们低效地要求同时将2D和3D模态作为输入，尽管3D点云固有的多视图属性已经包含2D模态；（2）输入2D模态会使重构学习不必要地依赖可见2D信息，阻碍3D几何表示学习。为应对这些挑战，我们提出一种3D到多视图学习器（Multi-View ML），其仅利用3D模态作为输入，并有效捕获3D点云中丰富的空间信息。具体而言，我们首先基于3D姿态在特征层级将3D点云投影为多视图2D图像。随后，我们引入两个组件：（1）3D到多视图自编码器，从3D特征和投影得到的2D特征重构点云和多视图图像；（2）多尺度多头（MSMH）注意力机制，通过不同尺度的注意力头促进每个解码器transformer块中的局部—全局信息交互。此外，我们提出一种新颖的两阶段自训练策略以对齐2D和3D表示。我们的方法在3D分类、部件分割和目标检测等多种下游任务上优于最先进的同类方法。
英文摘要：
Recently, multi-modal masked autoencoders (MAE) has been introduced in 3D self-supervised learning, offering enhanced feature learning by leveraging both 2D and 3D data to capture richer cross-modal representations. However, these approaches have two limitations: (1) they inefficiently require both 2D and 3D modalities as inputs, even though the inherent multi-view properties of 3D point clouds already contain 2D modality. (2) input 2D modality causes the reconstruction learning to unnecessarily rely on visible 2D information, hindering 3D geometric representation learning. To address these challenges, we propose a 3D to Multi-View Learner (Multi-View ML) that only utilizes 3D modalities as inputs and effectively capture rich spatial information in 3D point clouds. Specifically, we first project 3D point clouds to multi-view 2D images at the feature level based on 3D-based pose. Then, we introduce two components: (1) a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images from 3D and projected 2D features; (2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interactions in each decoder transformer block through attention heads at various scales. Additionally, a novel two-stage self-training strategy is proposed to align 2D and 3D representations. Our method outperforms state-of-the-art counterparts across various downstream tasks, including 3D classification, part segmentation, and object detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
