Tango3D: 向全局和局部2D-3D对应关系对齐迈进
Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence
AI总结：
本文提出Tango3D，一种统一密集对应和全局检索的3D基础模型，通过几何感知的2D视觉骨干网络和预训练的3D VAE将图像编码为2D片段，点云编码为3D标记，并映射到共享空间以实现局部像素-点对齐和全局语义对齐。
摘要：
现有的3D基础模型通常将点云对齐到冻结的视觉-语言空间（如CLIP），通过将3D形状压缩成全局向量实现强大的跨模态检索。然而，这种仅全局对齐的方法无法建立精细的像素-点对应关系。为了解决这个问题，我们提出了Tango3D，一种基础模型，它统一了密集对应和全局检索。我们使用一个几何感知的2D视觉骨干网络和一个预训练的3D VAE将图像编码为2D片段，并将点云编码为3D标记。这些被映射到一个共享空间中，以实现局部像素-点对齐和全局语义对齐。为了稳定密集和全局目标的联合学习，我们引入了三阶段渐进训练策略。实验表明，我们的模型成功实现了对象级别的像素-点对齐，同时保持了具有竞争力的全局检索能力，这种联合能力是现有3D基础模型所不具备的。通过建立精细的对齐特征空间，Tango3D将丰富的语义注入到纯粹的几何3D标记中，为广泛密集3D下游任务铺平了道路。
英文摘要：
Existing 3D foundation models typically align point clouds to frozen vision-language spaces like CLIP, which achieve strong cross-modal retrieval by compressing 3D shape into a global vector. However, this global-only alignment cannot establish fine-grained pixel-to-point correspondence. To solve this, we present Tango3D, a foundation model that unifies dense correspondence and global retrieval. We use a geometry-aware 2D visual backbone and a pretrained 3D VAE to encode images into 2D patches and point clouds into 3D tokens. These are mapped into a single shared space to achieve both local pixel-to-point alignment and global semantic alignment. To stabilize the joint learning of dense and global objectives, we introduce a three-stage progressive training strategy. Experiments show our model successfully achieves object-level pixel-to-point alignment while maintaining competitive global retrieval, a joint capability not offered by existing 3D foundation models. By establishing a fine-grained alignment feature space, Tango3D injects rich semantics into purely geometric 3D tokens, paving the way for a wide range of dense 3D downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
