arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.12308cs.CV
收藏

利用辐射场实现3D中的Segment Anything

Segment Anything in 3D with Radiance Fields

Shanghai Jiao Tong University(上海交通大学);Huawei Inc.(华为公司)

Jiazhong Cen, Jiemin Fang, Zanwei Zhou, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

AI总结:

本文提出SA3D,以辐射场连接多视图2D图像与3D空间,并通过掩码逆渲染和跨视图自提示迭代细化3D掩码,从而将SAM高效扩展到3D对象分割。

摘要:

Segment Anything Model(SAM)作为一种强大的视觉基础模型出现,可生成高质量的2D分割结果。本文旨在将SAM泛化到分割3D对象。我们没有复制在3D中代价高昂的数据采集和标注流程,而是设计了一种高效解决方案,利用辐射场作为廉价且现成的先验,将多视图2D图像与3D空间连接起来。我们将所提出的解决方案称为SA3D,即Segment Anything in 3D的缩写。使用SA3D时,用户只需在单个视图中为目标对象提供一个2D分割提示(例如粗略点),该提示用于通过SAM生成其对应的2D掩码。接下来,SA3D在各个视图之间交替执行掩码逆渲染和跨视图自提示,以迭代方式细化目标对象的3D掩码。对于一个视图,掩码逆渲染在辐射场所学习的密度分布引导下,将SAM获得的2D掩码投影到3D空间中,用于3D掩码细化;随后,跨视图自提示从针对新视图的、由不准确3D掩码渲染得到的2D掩码中自动提取可靠提示,作为SAM的输入。我们在实验中表明,SA3D能够适应各种场景,并在数秒内完成3D分割。我们的研究揭示了一种将2D分割模型能力提升到3D的潜在方法。我们的代码可在https://github.com/Jumpat/SegmentAnythingin3D获取。

英文摘要:

The Segment Anything Model (SAM) emerges as a powerful vision foundation model to generate high-quality 2D segmentation results. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the radiance field as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, short for Segment Anything in 3D. With SA3D, the user is only required to provide a 2D segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its corresponding 2D mask with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively refine the 3D mask of the target object. For one view, mask inverse rendering projects the 2D mask obtained by SAM into the 3D space with guidance of the density distribution learned by the radiance field for 3D mask refinement; Then, cross-view self-prompting extracts reliable prompts automatically as the input to SAM from the rendered 2D mask of the inaccurate 3D mask for a new view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within seconds. Our research reveals a potential methodology to lift the ability of a 2D segmentation model to 3D. Our code is available at https://github.com/Jumpat/SegmentAnythingin3D.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑