arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.02416cs.CVcs.AIcs.LGcs.RO

ODIN:用于2D和3D分割的统一模型

ODIN: A Single Model for 2D and 3D Segmentation

Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);Microsoft(微软公司)

Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W. Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, Katerina Fragkiadaki

AI总结:

ODIN提出一种交替融合2D视图内与3D跨视图信息的Transformer架构,统一处理2D图像和3D点云分割,在多个3D基准上达到最先进性能,并提升具身智能体的3D感知能力。

摘要:

在当代3D分割基准(如ScanNet)上,最先进的模型消费并标注数据集提供的3D点云,这些点云通过对感测到的多视图RGB-D图像进行后处理获得。这些模型通常在领域内训练,放弃了大规模2D预训练,并且优于那些对带姿态的RGB-D多视图图像进行特征化的替代方法。消费带姿态图像的方法与消费后处理3D点云的方法之间的性能差距,助长了2D和3D感知需要不同模型架构的信念。在本文中,我们挑战这一观点,并提出ODIN(全维度实例分割),一个能够同时分割和标注2D RGB图像与3D点云的模型,采用一种在2D视图内和3D跨视图信息融合之间交替的Transformer架构。我们的模型通过所涉及令牌的位置编码来区分2D和3D特征操作,这些编码为2D补丁令牌捕获像素坐标,为3D特征令牌捕获3D坐标。ODIN在ScanNet200、Matterport3D和AI2THOR 3D实例分割基准上取得了最先进的性能,并在ScanNet、S3DIS和COCO上取得了有竞争力的性能。当使用感测到的3D点云代替从3D网格采样的点云时,它以较大优势超越了所有先前的工作。当用作可指令具身智能体架构中的3D感知引擎时,它在TEACh动作-对话基准上创下了新的最先进水平。我们的代码和检查点可在项目网站(https://odin-seg.github.io)上找到。

英文摘要:

State-of-the-art models on contemporary 3D segmentation benchmarks like ScanNet consume and label dataset-provided 3D point clouds, obtained through post processing of sensed multiview RGB-D images. They are typically trained in-domain, forego large-scale 2D pre-training and outperform alternatives that featurize the posed RGB-D multiview images instead. The gap in performance between methods that consume posed images versus post-processed 3D point clouds has fueled the belief that 2D and 3D perception require distinct model architectures. In this paper, we challenge this view and propose ODIN (Omni-Dimensional INstance segmentation), a model that can segment and label both 2D RGB images and 3D point clouds, using a transformer architecture that alternates between 2D within-view and 3D cross-view information fusion. Our model differentiates 2D and 3D feature operations through the positional encodings of the tokens involved, which capture pixel coordinates for 2D patch tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It outperforms all previous works by a wide margin when the sensed 3D point cloud is used in place of the point cloud sampled from 3D mesh. When used as the 3D perception engine in an instructable embodied agent architecture, it sets a new state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and checkpoints can be found at the project website (https://odin-seg.github.io).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑