3D scene understanding for robotic applications exhibits a unique set of requirements including real-time inference, object-centric latent representation learning, accurate 6D pose estimation and 3D reconstruction of objects. Current methods for scene understanding typically rely on a combination of trained models paired with either an explicit or learnt volumetric representation, all of which have their own drawbacks and limitations. We introduce DreamUp3D, a novel Object-Centric Generative Model (OCGM) designed explicitly to perform inference on a 3D scene informed only by a single RGB-D image. DreamUp3D is a self-supervised model, trained end-to-end, and is capable of segmenting objects, providing 3D object reconstructions, generating object-centric latent representations and accurate per-object 6D pose estimates. We compare DreamUp3D to baselines including NeRFs, pre-trained CLIP-features, ObSurf, and ObPose, in a range of tasks including 3D scene reconstruction, object matching and object pose estimation. Our experiments show that our model outperforms all baselines by a significant margin in real-world scenarios displaying its applicability for 3D scene understanding tasks while meeting the strict demands exhibited in robotics applications.
CubifyGS: Object-Centric 3D Gaussian Splatting for Lifelong Dynamic Scene Maintenance
School of Automation, Beijing Institute of Technology, Beijing, China(北京理工大学自动化学院,北京,中国);Guangzhou Saite Intelligent Technology Co., Ltd.(广州赛泰智能科技有限公司)
Hi-Dyna Graph: Hierarchical Dynamic Scene Graph for Robotic Autonomy in Human-Centric Environments
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学脑启发式智能科学研究院)
GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes
KTH Royal Institute of Technology(瑞典皇家理工学院);University of Stuttgart(斯图加特大学);Max Planck Research School for Intelligent Systems(马克斯·普朗克智能系统研究所);Google Research(谷歌研究院);TU Munich(慕尼黑工业大学)
EvObj: Learning Evolving Object-centric Representations for 3D Instance Segmentation without Scene Supervision
Shenzhen Research Institute, The Hong Kong Polytechnic University(深圳研究 institute,香港理工大学);vLAR Group, The Hong Kong Polytechnic University(vLAR 团队,香港理工大学)
Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations
Autonomous Intelligent Systems(自主智能系统);Computer Science Institute VI(计算机科学研究所VI);Intelligent Systems and Robotics(智能系统与机器人);Center for Robotics(机器人中心);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔人工智能学习与智能研究所);University of Bonn, Germany(波恩大学,德国)