arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.19782cs.CV
收藏

3D 人类开放词汇语义部件分割

Open-Vocabulary Semantic Part Segmentation of 3D Human

University of California, San Diego(加利福尼亚大学圣迭戈分校);Qualcomm(高通公司)

Keito Suzuki, Bang Du, Girish Krishnan, Kunyao Chen, Runfa Blark Li, Truong Nguyen

AI总结:

提出首个开放词汇3D人类部件分割方法,利用SAM生成多视图提案和HumanCLIP统一嵌入,结合MaskFusion模块,无需复杂投票机制,大幅优于现有方法并适用于多种3D表示。

摘要:

3D 部件分割在 3D 视觉和 AR/VR 领域仍是一个开放问题。由于 3D 标注数据有限,传统的监督分割方法难以泛化到未见过的形状和类别。近年来,视觉-语言模型零样本能力的进步带来了开放世界 3D 分割方法的激增。虽然这些方法在 3D 场景或物体上显示出有前景的结果,但它们不能很好地泛化到 3D 人类。在本文中,我们提出了第一个能够处理 3D 人类的开放词汇分割方法。我们的框架可以根据文本提示将人类类别分割成所需的细粒度部件。我们设计了一个简单的分割流程,利用 SAM 在 2D 中生成多视图提案,并提出了一种新颖的 HumanCLIP 模型来为视觉和文本输入创建统一的嵌入。与现有的预训练 CLIP 模型相比,HumanCLIP 模型为以人为中心的内容生成了更准确的嵌入。我们还设计了一个简单而有效的 MaskFusion 模块,该模块无需复杂的投票和分组机制即可将多视图特征分类并融合成 3D 语义掩码。解耦掩码提案和文本输入的设计也显著提高了每次提示推理的效率。在多个 3D 人类数据集上的实验结果表明,我们的方法大幅优于当前最先进的开放词汇 3D 分割方法。此外,我们表明我们的方法可以直接应用于各种 3D 表示,包括网格、点云和 3D 高斯泼溅。

英文摘要:

3D part segmentation is still an open problem in the field of 3D vision and AR/VR. Due to limited 3D labeled data, traditional supervised segmentation methods fall short in generalizing to unseen shapes and categories. Recently, the advancement in vision-language models' zero-shot abilities has brought a surge in open-world 3D segmentation methods. While these methods show promising results for 3D scenes or objects, they do not generalize well to 3D humans. In this paper, we present the first open-vocabulary segmentation method capable of handling 3D human. Our framework can segment the human category into desired fine-grained parts based on the textual prompt. We design a simple segmentation pipeline, leveraging SAM to generate multi-view proposals in 2D and proposing a novel HumanCLIP model to create unified embeddings for visual and textual inputs. Compared with existing pre-trained CLIP models, the HumanCLIP model yields more accurate embeddings for human-centric contents. We also design a simple-yet-effective MaskFusion module, which classifies and fuses multi-view features into 3D semantic masks without complex voting and grouping mechanisms. The design of decoupling mask proposals and text input also significantly boosts the efficiency of per-prompt inference. Experimental results on various 3D human datasets show that our method outperforms current state-of-the-art open-vocabulary 3D segmentation methods by a large margin. In addition, we show that our method can be directly applied to various 3D representations including meshes, point clouds, and 3D Gaussian Splatting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑