arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.02157cs.CVcs.AI
收藏

用语言分割任意三维物体

Segment Any 3D Object with Language

Korea University(高丽大学);National University of Singapore(新加坡国立大学)

Seungjun Lee, Yuyang Zhao, Gim Hee Lee

AI总结:

本文提出SOLE框架,通过多模态融合网络直接从三维点云生成语义相关掩码,并引入三种多模态关联监督,在多个基准上大幅提升开放词汇三维实例分割性能,接近全监督方法。

摘要:

本文研究基于自由形式语言指令的开放词汇三维实例分割(OV-3DIS)。早期工作仅依赖标注的基础类别进行训练,对未见新类别的泛化能力有限。近期工作通过生成类别无关掩码或将二维泛化掩码投影到三维来缓解对新类别泛化能力差的问题,但忽视了语义或几何信息,导致性能欠佳。相比之下,直接从三维点云生成可泛化且语义相关的掩码将获得更优结果。本文提出用语言分割任意三维物体(SOLE),这是一个语义与几何感知的视觉-语言学习框架,通过直接从三维点云生成语义相关掩码而具备强泛化能力。具体而言,我们提出一个多模态融合网络,在骨干网络和解码器中融入多模态语义。此外,为使三维分割模型与各种语言指令对齐并提升掩码质量,我们引入三种类型的多模态关联作为监督。我们的SOLE在ScanNetv2、ScanNet200和Replica基准上大幅超越先前方法,且尽管训练中缺少类别标注,结果甚至接近全监督对应方法。此外,大量定性结果展示了SOLE对语言指令的多样性适应能力。

英文摘要:

In this paper, we investigate Open-Vocabulary 3D Instance Segmentation (OV-3DIS) with free-form language instructions. Earlier works that rely on only annotated base categories for training suffer from limited generalization to unseen novel categories. Recent works mitigate poor generalizability to novel categories by generating class-agnostic masks or projecting generalized masks from 2D to 3D, but disregard semantic or geometry information, leading to sub-optimal performance. Instead, generating generalizable but semantic-related masks directly from 3D point clouds would result in superior outcomes. In this paper, we introduce Segment any 3D Object with LanguagE (SOLE), which is a semantic and geometric-aware visual-language learning framework with strong generalizability by generating semantic-related masks directly from 3D point clouds. Specifically, we propose a multimodal fusion network to incorporate multimodal semantics in both backbone and decoder. In addition, to align the 3D segmentation model with various language instructions and enhance the mask quality, we introduce three types of multimodal associations as supervision. Our SOLE outperforms previous methods by a large margin on ScanNetv2, ScanNet200, and Replica benchmarks, and the results are even close to the fully-supervised counterpart despite the absence of class annotations in the training. Furthermore, extensive qualitative results demonstrate the versatility of our SOLE to language instructions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑