Segment Anything Model(SAM)作为一种强大的视觉基础模型出现,可生成高质量的2D分割结果。本文旨在将SAM泛化到分割3D对象。我们没有复制在3D中代价高昂的数据采集和标注流程,而是设计了一种高效解决方案,利用辐射场作为廉价且现成的先验,将多视图2D图像与3D空间连接起来。我们将所提出的解决方案称为SA3D,即Segment Anything in 3D的缩写。使用SA3D时,用户只需在单个视图中为目标对象提供一个2D分割提示(例如粗略点),该提示用于通过SAM生成其对应的2D掩码。接下来,SA3D在各个视图之间交替执行掩码逆渲染和跨视图自提示,以迭代方式细化目标对象的3D掩码。对于一个视图,掩码逆渲染在辐射场所学习的密度分布引导下,将SAM获得的2D掩码投影到3D空间中,用于3D掩码细化;随后,跨视图自提示从针对新视图的、由不准确3D掩码渲染得到的2D掩码中自动提取可靠提示,作为SAM的输入。我们在实验中表明,SA3D能够适应各种场景,并在数秒内完成3D分割。我们的研究揭示了一种将2D分割模型能力提升到3D的潜在方法。我们的代码可在https://github.com/Jumpat/SegmentAnythingin3D获取。
英文摘要:
The Segment Anything Model (SAM) emerges as a powerful vision foundation model to generate high-quality 2D segmentation results. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the radiance field as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, short for Segment Anything in 3D. With SA3D, the user is only required to provide a 2D segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its corresponding 2D mask with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively refine the 3D mask of the target object. For one view, mask inverse rendering projects the 2D mask obtained by SAM into the 3D space with guidance of the density distribution learned by the radiance field for 3D mask refinement; Then, cross-view self-prompting extracts reliable prompts automatically as the input to SAM from the rendered 2D mask of the inaccurate 3D mask for a new view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within seconds. Our research reveals a potential methodology to lift the ability of a 2D segmentation model to 3D. Our code is available at https://github.com/Jumpat/SegmentAnythingin3D.
Scene-Generalizable Interactive Segmentation of Radiance Fields
Harbin Institute of Technology(哈尔滨工业大学);Y-tech, Kuaishou Technology(快手科技Y-tech);Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东省新型安全智能技术重点实验室);Dartmouth College(达特茅斯学院)
Polyp-DAM: Polyp segmentation via depth anything model
Nanjing University of Science and Technology(南京理工大学);University of Science and Technology of China(中国科学技术大学);Sun Yat-sen University(中山大学);National University of Singapore(新加坡国立大学)
Nanyang Technological University(南洋理工大学);SenseTime Research(商汤科技研究院);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
TASAM: Terrain-and-Aware Segment Anything Model for Temporal-Scale Remote Sensing Segmentation
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Wuhan University(武汉大学);Dalian University of Technology(大连理工大学);University of Chinese Academy of Sciences(中国科学院大学)
Biomedical SAM 2: Segment Anything in Biomedical Images and Videos
University of Georgia(佐治亚大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Lehigh University(理海大学);Northeastern University(东北大学);University of Notre Dame(圣母大学)
Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping
Arizona State University(亚利桑那州立大学);Woodwell Climate Research Center(伍德韦尔气候研究中心);University of Connecticut(康涅狄格大学);U.S. Geological Survey(美国地质调查局);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);University of Illinois at Urbana-Champaign(伊利诺伊大学香槟分校)
本文评估了以 Meta SAM 为代表的视觉基础模型在永久冻土地貌分割中的表现,通过多种提示策略和微调测试其零样本能力与领域适应性,并指出其在挑战性地理空间任务中仍需改进。
SAMamba3D: adapting Segment Anything for generalizable 3D segmentation of multiphase pore-scale images
Department of Earth Science and Engineering, Imperial College London(帝国理工学院伦敦地球科学与工程系);College of Artificial Intelligence, China University of Petroleum (Beijing)(中国石油大学(北京)人工智能学院);College of Petroleum Engineering, China University of Petroleum (Beijing)(中国石油大学(北京)石油工程学院)
Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation
The Chinese University of Hong Kong(香港中文大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与XR研究所)
提出 CPC-SAM,在双分支框架中通过 SAM 交叉提示和提示一致性正则化利用标注与未标注数据,显著提升半监督医学图像分割性能。