arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.06403cs.CV
收藏

PointSeg:基于基础模型的3D场景分割免训练范式

PointSeg: A Training-Free Paradigm for 3D Scene Segmentation via Foundation Models

Tencent Youtu Lab(腾讯优图实验室)

Qingdong He, Jinlong Peng, Zhengkai Jiang, Xiaobin Hu, Jiangning Zhang

AI总结:

提出免训练范式PointSeg,利用视觉基础模型解决3D场景分割问题,通过双分支提示学习与双向匹配生成3D提示,在多个数据集上显著超越现有免训练和基于训练的专用方法。

摘要:

视觉基础模型近期的成功在2D感知任务中展现出了令人期待的性能。然而,由于数据集有限,直接训练3D基础网络十分困难,且现有基础模型能否无缝提升至3D空间仍有待探索。本文提出PointSeg,一种新颖的免训练范式,利用现成的视觉基础模型解决3D场景感知任务。PointSeg通过获取准确的3D提示来对齐跨帧的对应像素,从而在3D场景中分割任意物体。具体而言,我们设计了一个双分支提示学习结构来构建3D点-框提示对,并结合双向匹配策略以生成准确的点提示和提议提示。随后,在与不同的视觉基础模型协作时,我们自适应地执行迭代后细化。此外,我们设计了一种亲和力感知的合并算法来改进最终的集成掩码。PointSeg在无需训练的情况下,在多个数据集上展现了出色的分割性能。具体来说,我们的方法在ScanNet、ScanNet++和KITTI-360数据集上,分别以14.1%、12.3%和12.6%的mAP显著超越了最先进的免训练专用模型。不仅如此,PointSeg能够与多种基础模型结合,甚至在多个数据集上以3.4%-5.4%的mAP超越了基于训练的专用方法,成为一种有效的通用模型。

英文摘要:

Recent success of vision foundation models have shown promising performance for the 2D perception tasks. However, it is difficult to train a 3D foundation network directly due to the limited dataset and it remains under explored whether existing foundation models can be lifted to 3D space seamlessly. In this paper, we present PointSeg, a novel training-free paradigm that leverages off-the-shelf vision foundation models to address 3D scene perception tasks. PointSeg can segment anything in 3D scene by acquiring accurate 3D prompts to align their corresponding pixels across frames. Concretely, we design a two-branch prompts learning structure to construct the 3D point-box prompts pairs, combining with the bidirectional matching strategy for accurate point and proposal prompts generation. Then, we perform the iterative post-refinement adaptively when cooperated with different vision foundation models. Moreover, we design a affinity-aware merging algorithm to improve the final ensemble masks. PointSeg demonstrates impressive segmentation performance across various datasets, all without training. Specifically, our approach significantly surpasses the state-of-the-art specialist training-free model by 14.1$\%$, 12.3$\%$, and 12.6$\%$ mAP on ScanNet, ScanNet++, and KITTI-360 datasets, respectively. On top of that, PointSeg can incorporate with various foundation models and even surpasses the specialist training-based methods by 3.4$\%$-5.4$\%$ mAP across various datasets, serving as an effective generalist model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑