3D- OES：视点不变的对象分解环境模拟器
3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
摘要：
我们提出了一个动作条件动力学模型，预测场景变化所造成的对象和代理的相互作用，在一个视点不变的3D神经场景表示空间，从RGB-D视频推断。在这个3D特征空间中，对象不会相互干扰，并且它们的外观会随着时间的推移和视点的变化而保持不变。这使得我们的模型可以通过简单地基于累积对象运动预测“移动”3D对象特征来预测未来很长一段时间的场景。对象运动预测由图神经网络计算，该图神经网络对从3D神经场景表示中提取的对象特征进行操作。我们的模型的模拟可以由神经渲染器从任何期望的视角解码成2D图像视图，这有助于我们潜在的3D模拟空间的可解释性。我们表明，我们的模型在不同数量和外观的交互对象以及相机的观点，以及概括其预测优于现有的2D和3D动态模型。我们进一步展示了仿真到现实迁移的学习动力学，通过应用我们的模型训练，仅在模拟基于模型的控制下，推动物体到所需的位置下，在一个真正的机器人设置混乱
英文摘要：
We propose an action-conditioned dynamics model that predicts scene changes caused by object and agent interactions in a viewpoint-invariant 3D neural scene representation space, inferred from RGB-D videos. In this 3D feature space, objects do not interfere with one another and their appearance persists over time and across viewpoints. This permits our model to predict future scenes long in the future by simply "moving" 3D object features based on cumulative object motion predictions. Object motion predictions are computed by a graph neural network that operates over the object features extracted from the 3D neural scene representation. Our model's simulations can be decoded by a neural renderer into2D image views from any desired viewpoint, which aids the interpretability of our latent 3D simulation space. We show our model generalizes well its predictions across varying number and appearances of interacting objects as well as across camera viewpoints, outperforming existing 2D and 3D dynamics models. We further demonstrate sim-to-real transfer of the learnt dynamics by applying our model trained solely in simulation to model-based control for pushing objects to desired locations under clutter on a real robotic setup
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
