arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.01558cs.CVcs.RO
收藏

PartSLIP:通过预训练图像-语言模型实现三维点云的低样本部件分割

PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models

UC San Diego(加州大学圣地亚哥分校);Qualcomm AI Research(高通人工智能研究院)

Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, Hao Su

AI总结:

本文提出PartSLIP方法,利用预训练图像-语言模型GLIP和二维到三维标签提升算法,实现三维点云的低样本部件分割,在PartNet等数据集上达到优异性能。

摘要:

可泛化的三维部件分割在视觉和机器人领域重要但具有挑战性。通过传统监督方法训练深度模型需要具有细粒度部件标注的大规模三维数据集,这些数据集的收集成本高昂。本文探索了一种利用预训练图像-语言模型GLIP(一种在开放词汇二维检测中表现优异的模型)实现三维点云低样本部件分割的替代方法。我们通过基于GLIP的点云渲染部件检测以及一种新颖的二维到三维标签提升算法,将丰富的知识从二维迁移到三维。我们还利用多视角三维先验和少样本提示调优来显著提升性能。在PartNet和PartNet-Mobility数据集上的广泛评估表明,我们的方法实现了出色的零样本三维部件分割。我们的少样本版本不仅大幅优于现有少样本方法,而且与全监督方法相比也取得了极具竞争力的结果。此外,我们证明了我们的方法可以直接应用于iPhone扫描的点云,而无需显著的域差距。

英文摘要:

Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP, which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑