arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16308cs.RO
收藏

DreamUp3D:面向单视图3D场景理解与真实到仿真迁移的以对象为中心的生成模型

DreamUp3D: Object-Centric Generative Models for Single-View 3D Scene Understanding and Real-to-Sim Transfer

Applied AI Lab, Oxford Robotics Institute(牛津机器人研究所应用人工智能实验室)

Yizhe Wu, Haitz Sáez de Ocáriz Borde, Jack Collins, Oiwi Parker Jones, Ingmar Posner

AI总结:

DreamUp3D提出一种以对象为中心的生成模型,仅从单张RGB-D图像实现3D场景分割、重建、潜在表示和6D姿态估计,并在真实场景中显著超越NeRF、CLIP、ObSurf和ObPose等基线。

摘要:

面向机器人应用的3D场景理解展现出一组独特需求,包括实时推理、以对象为中心的潜在表示学习、精确的6D姿态估计以及对象的3D重建。当前场景理解方法通常依赖训练模型与显式或学习得到的体积表示的组合,而这些方法各有其缺陷与局限。我们提出DreamUp3D,一种新颖的以对象为中心的生成模型(Object-Centric Generative Model, OCGM),专门设计用于仅依据单张RGB-D图像对3D场景进行推理。DreamUp3D是一种自监督模型,采用端到端训练,能够分割对象、提供3D对象重建、生成以对象为中心的潜在表示,并给出每个对象的精确6D姿态估计。我们将DreamUp3D与包括NeRF、预训练CLIP特征、ObSurf和ObPose在内的基线方法在一系列任务中进行比较,包括3D场景重建、对象匹配和对象姿态估计。实验表明,我们的模型在真实世界场景中以显著优势超越所有基线方法,展示了其在3D场景理解任务中的适用性,同时满足机器人应用中展现的严格要求。

英文摘要:

3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of trained models paired with either an explicit or learnt volumetric representation, all of which have their own drawbacks and limitations. We introduce DreamUp3D, a novel Object-Centric Generative Model (OCGM) designed explicitly to perform inference on a 3D scene informed only by a single RGB-D image. DreamUp3D is a self-supervised model, trained end-to-end, and is capable of segmenting objects, providing 3D object reconstructions, generating object-centric latent representations and accurate per-object 6D pose estimates. We compare DreamUp3D to baselines including NeRFs, pre-trained CLIP-features, ObSurf, and ObPose, in a range of tasks including 3D scene reconstruction, object matching and object pose estimation. Our experiments show that our model outperforms all baselines by a significant margin in real-world scenarios displaying its applicability for 3D scene understanding tasks while meeting the strict demands exhibited in robotics applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑