arXiv 2401.094132024-01-17cs.CV

POP-3D：从图像进行开放词汇3D占用预测

POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

CIIRC CTU in Prague（布拉格捷克理工大学捷克信息、机器人与控制论研究所）；FEE CTU in Prague（布拉格捷克理工大学电气工程学院）

Antonin Vobecky, Oriane Siméoni, David Hurych, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Josef Sivic

备注：: accepted to NeurIPS 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

POP-3D提出一种从2D图像预测开放词汇3D语义占用图的架构，结合三模态自监督学习，无需3D语言标注即可实现零样本3D分割及自由形式语言查询的3D定位与检索。

摘要：

我们描述了一种从输入2D图像预测开放词汇3D语义体素占用图的方法，目标是实现自由形式语言查询的3D定位、分割和检索。这是一个具有挑战性的问题，因为存在2D-3D歧义以及目标任务的开放词汇特性，而在3D中获取带标注的训练数据十分困难。本工作的贡献有三方面。第一，我们设计了一种新的用于开放词汇3D语义占用预测的模型架构。该架构由一个2D-3D编码器以及占用预测头和3D-语言头组成。输出是一个密集的3D定位语言嵌入体素图，能够支持一系列开放词汇任务。第二，我们开发了一种三模态自监督学习算法，利用三种模态：（i）图像，（ii）语言和（iii）LiDAR点云，并能够使用强大的预训练视觉-语言模型训练所提出的架构，而无需任何3D人工语言标注。最后，我们在多个开放词汇任务上定量展示了所提出模型的优势：使用现有数据集进行零样本3D语义分割；使用我们作为nuScenes扩展提出的小型数据集进行自由形式语言查询的3D定位和检索。您可以在https://vobecant.github.io/POP3D找到项目页面。

英文摘要：

We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The contributions of this work are three-fold. First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes. You can find the project page here https://vobecant.github.io/POP3D.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。