RORA：带关节运动的真实物体重建
RORA: Realistic Object Reconstruction with Articulation
浏览论文内容
中文总结 AI 辅助
本研究提出RORA端到端管线，可从单个静态物体视频输入重建带精确关节运动的仿真就绪资产，在PartNet-Mobility-V0数据集和真实物体上取得良好效果，可用于机器人学习的实时灵巧手操作任务。
中文摘要 AI 辅助
通过NeRF、3D高斯溅射（3DGS）等真实视觉表示将现实环境复制到仿真中，已成为缩小机器人学习中现实-仿真差距的有效策略。然而，在现实转仿真过程中实现物体关节运动仍是一项具有挑战性的任务。现有的基于运动跟踪或学习的关节运动方法，在具有多个关节的复杂运动学结构上成功率较低。此外，这些方法需要扫描物体的动态运动，这使得重建过程更加复杂。在本研究中，我们提出了首个端到端管线，可通过基于建议的人在回路过程，从单个静态物体视频输入中重建出带有精确关节运动的仿真就绪资产。我们的方法导出了一种混合表示，结合了用于照片级真实感渲染的3DGS和用于物理交互的基于网格的几何。在重建过程中，我们的管线执行凸分解，随后进行用户分组以实现直观的部件分割，接着将3D高斯绑定到对应的网格部件。自动关节建议算法随后从局部边界几何计算候选关节轴，并将其呈现给用户以高效进行关节资产重建。我们已证明，我们的方法在PartNet-Mobility-V0数据集和真实物体上实现了精确的关节运动结果。此外，我们展示了我们框架在机器人学习中的潜在应用，将重建的资产部署到Unreal Engine和NVIDIA Isaac Sim中，展示了实时灵巧手操作任务。
英文摘要
Replicating real-world environments into simulation by realistic visual representation like NeRF and 3D Gaussian Splatting (3DGS) has emerged as an effective strategy to reduce the sim-to-real gap in robot learning. However, implementing object articulation during the real-to-sim process is still a challenging task. Existing motion tracking or learning based articulation methods shows low success rates on complex kinematic structures having multiple joints. Furthermore, those methods require scan of dynamic motion of objects, which makes reconstruction process much complicated. In this work, we propose the first end-to-end pipeline that reconstructs simulation-ready assets with accurate articulation from a single static object video input through suggestion based human-in-the-loop process. Our approach exports a hybrid representation combining 3DGS for photorealistic rendering and mesh-based geometry for physical interaction. In the reconstruction process, our pipeline performs convex decomposition followed by user grouping for intuitive part segmentation, subsequently binding 3D Gaussians to the corresponding mesh parts. An Automatic Joint Suggestion Algorithm then calculates candidate joint axes from local boundary geometries and presents them to users for efficient articulated asset reconstruction. We have shown that our method achieves precise articulation results on partnet-mobility-v0 dataset and real objects. Additionally we presented a potential usage of our framework on robot learning, deploying the reconstructed assets in Unreal Engine and NVIDIA Isaac Sim, demonstrating real-time dexterous hand manipulation tasks.
发表机构
- Seoul National University（首尔大学）
- DGIST（大邱庆北科学技术院）
机构由 AI 辅助整理，请以论文原文为准。
