3D 部件分割在 3D 视觉和 AR/VR 领域仍是一个开放问题。由于 3D 标注数据有限,传统的监督分割方法难以泛化到未见过的形状和类别。近年来,视觉-语言模型零样本能力的进步带来了开放世界 3D 分割方法的激增。虽然这些方法在 3D 场景或物体上显示出有前景的结果,但它们不能很好地泛化到 3D 人类。在本文中,我们提出了第一个能够处理 3D 人类的开放词汇分割方法。我们的框架可以根据文本提示将人类类别分割成所需的细粒度部件。我们设计了一个简单的分割流程,利用 SAM 在 2D 中生成多视图提案,并提出了一种新颖的 HumanCLIP 模型来为视觉和文本输入创建统一的嵌入。与现有的预训练 CLIP 模型相比,HumanCLIP 模型为以人为中心的内容生成了更准确的嵌入。我们还设计了一个简单而有效的 MaskFusion 模块,该模块无需复杂的投票和分组机制即可将多视图特征分类并融合成 3D 语义掩码。解耦掩码提案和文本输入的设计也显著提高了每次提示推理的效率。在多个 3D 人类数据集上的实验结果表明,我们的方法大幅优于当前最先进的开放词汇 3D 分割方法。此外,我们表明我们的方法可以直接应用于各种 3D 表示,包括网格、点云和 3D 高斯泼溅。
英文摘要:
3D part segmentation is still an open problem in the field of 3D vision and AR/VR. Due to limited 3D labeled data, traditional supervised segmentation methods fall short in generalizing to unseen shapes and categories. Recently, the advancement in vision-language models' zero-shot abilities has brought a surge in open-world 3D segmentation methods. While these methods show promising results for 3D scenes or objects, they do not generalize well to 3D humans. In this paper, we present the first open-vocabulary segmentation method capable of handling 3D human. Our framework can segment the human category into desired fine-grained parts based on the textual prompt. We design a simple segmentation pipeline, leveraging SAM to generate multi-view proposals in 2D and proposing a novel HumanCLIP model to create unified embeddings for visual and textual inputs. Compared with existing pre-trained CLIP models, the HumanCLIP model yields more accurate embeddings for human-centric contents. We also design a simple-yet-effective MaskFusion module, which classifies and fuses multi-view features into 3D semantic masks without complex voting and grouping mechanisms. The design of decoupling mask proposals and text input also significantly boosts the efficiency of per-prompt inference. Experimental results on various 3D human datasets show that our method outperforms current state-of-the-art open-vocabulary 3D segmentation methods by a large margin. In addition, we show that our method can be directly applied to various 3D representations including meshes, point clouds, and 3D Gaussian Splatting.
Nanyang Technological University(南洋理工大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);University of Ottawa(渥太华大学);Carnegie Mellon University(卡内基梅隆大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Bosch Research North America(博世北美研究院);Bosch Center for Artificial Intelligence (BCAI)(博世人工智能中心);The Ohio State University(俄亥俄州立大学);University of California Davis(加州大学戴维斯分校)
Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation
The Good AI Lab(好AI实验室);ETH Zurich(苏黎世联邦理工学院);AMD Silo AI;Google(谷歌);Technical University of Munich(慕尼黑技术大学);INSAIT, Sofia University, St. Kliment Ohridski(INSAIT,索菲亚大学,圣克莱孟·奥赫里茨基);University of Bologna(博洛尼亚大学)
School of Electronic Information, Wuhan University of Science and Technology(武汉科技大学电子信息学院);State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(北京邮电大学网络与交换技术国家重点实验室);Oriental Space Port Research Institute(东方航天港研究院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc(字节跳动公司);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
A Simple Framework for Open-Vocabulary Segmentation and Detection
The Hong Kong University of Science and Technology(香港科技大学);International Digital Economy Academy (IDEA)(国际数字经济研究院(IDEA));University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学);Microsoft Research at Redmond(微软雷德蒙德研究院)
Renovating Names in Open-Vocabulary Segmentation Benchmarks
Bosch IoC Lab(博世物联网计算实验室);University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Autonomous Vision Group, University of Tübingen(蒂宾根大学自主视觉组);ETH Zurich(苏黎世联邦理工学院);MPI for Intelligent Systems, Tübingen(蒂宾根智能系统马克斯·普朗克研究所);Bosch Center for Artificial Intelligence(博世人工智能中心)
Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models
Artificial Intelligence Creative Research Lab, ETRI University of Science(人工智能创意研究实验室,ETRI大学);Field Robotics Research Section, ETRI University of Science(机器人领域研究部,ETRI大学);Artificial Intelligence Creative Research Lab, ETRI Daejeon, South Korea(人工智能创意研究实验室,ETRI大田,韩国)