RoboSnap：用于可泛化机器人学习与评估的一次性真实到模拟场景生成
RoboSnap: One-Shot Real-to-Sim Scene Generation for Generalizable Robot Learning and Evaluation
浏览论文内容
中文总结 AI 辅助
针对构建物理稳定且视觉逼真场景缓慢昂贵的问题，提出RoboSnap框架，通过分层设计将单张RGB图像转为模拟场景，实验证明其能实现轨迹重放等，还引入DROID-Sim数据集，凸显真实到模拟方法对机器人学习评估的价值。
中文摘要 AI 辅助
将真实世界场景恢复为交互式模拟环境可实现可泛化机器人学习和可重复的策略评估。然而，构建物理稳定且视觉逼真的场景仍然缓慢且昂贵。本文提出RoboSnap，一个将单张RGB图像转换为可用于模拟的场景的真实到模拟框架。关键思想是分层设计，将物理关键交互区域与周围视觉上下文分开。实验表明，RoboSnap在恢复的场景中实现了可靠的轨迹重放，支持用于策略训练的特定任务合成数据生成，并为策略评估产生有意义的模拟-真实相关性。为进一步支持真实到模拟研究，引入了DROID-Sim数据集。广泛实验表明，真实到模拟方法的价值不仅在于高保真视觉重建，还在于将真实环境转变为机器人学习和评估的可重复使用基础设施。
英文摘要
Recovering real-world scenes as interactive simulation environments can enable generalizable robot learning and reproducible policy evaluation. However, constructing scenes that are both physically stable and visually faithful remains slow and expensive. In this work, we present RoboSnap, a real-to-sim framework that turns a single RGB image into a simulation-ready scene. The key idea is a layered design that separates the physics-critical interaction area from the surrounding visual context: collision-aware foreground assets are refined for stable robot interaction, while a 3D Gaussian splatting visual layer preserves faithful background appearance under novel views. Experiments on DROID scenes and real-robot tasks show that RoboSnap achieves reliable trajectory replay in the recovered scenes, supports task-specific synthetic data generation for policy training, and yields meaningful sim-real correlation for policy evaluation. To further support real-to-sim research, we introduce DROID-Sim, a real-to-sim companion dataset constructed from 564 real-world scenes in DROID. Extensive experiments suggest that the value of real-to-sim methods lies not only in high-fidelity visual reconstruction, but in turning real environments into reusable infrastructure for robot learning and evaluation.
发表机构
- Shanghai AI Laboratory（上海人工智能实验室）
- Shanghai Jiao Tong University（上海交通大学）
- Zhejiang University（浙江大学）
- Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。
