发表机构
Shanghai Artificial Intelligence Laboratory; Shanghai Jiao Tong University; Zhejiang University; University of Science and Technology of China; The Chinese University of Hong Kong; The University of Hong Kong（上海人工智能实验室; 上海交通大学; 浙江大学; 中国科学技术大学; 香港中文大学; 香港大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Real2Gym提出智能体式Real2Sim2Real框架，将视频演示转为交互仿真环境，提炼可复用技能并迁移至真实机器人，显著提升成功率并降低执行成本。
AI 中文摘要
现实世界的视频提供了丰富的操作演示，但将其转化为可复用的机器人技能需要视觉对齐的环境、可执行的物理交互以及从经验中学习的机制。我们提出了Real2Gym，一个智能体式的Real2Sim2Real框架，它将人类和机器人的演示转化为交互式仿真健身房，并将仿真中习得的技能迁移到物理机器人上。Real2Sim模块重建可编辑场景，将物体和相机与输入对齐，通过原生物理执行验证演示或重定向的动作，并生成带有动作可行性检查的任务条件变体。在这些环境中，智能体为操作阶段生成可执行代码，观察其结果，并将成功的尝试和失败提炼为可复用的任务流程、物体相对运动以及恢复策略。通过共享的感知与控制接口，这些技能指导后续在仿真和真实机器人上的执行，动作根据当前观察进行调整，且不更新底层模型权重。广泛评估表明，Real2Gym能够实现高保真仿真环境重建，在这些环境中，其成功率比GPT-6 Astra Direct Mode高出16.7%，策略执行令牌数减少约74.9%，同时在真实Franka机器人的四项任务中，物理机器人执行成功率高出33.3%。
英文摘要
Real-world videos provide rich demonstrations of manipulation, but turning them into reusable robot skills requires visually aligned environments, executable physical interactions, and mechanisms for learning from experience. We introduce Real2Gym, an agentic Real2Sim2Real framework that turns human and robot demonstrations into interactive simulation gyms and brings skills acquired in simulation to physical robots. The Real2Sim module reconstructs editable scenes, aligns objects and cameras with the input, validates demonstrated or retargeted actions through native physics execution, and generates task-conditioned variations with action-feasibility checks. Within these environments, the agent generates executable code for manipulation stages, observes their outcomes, and distills successful attempts and failures into reusable task procedures, object-relative motions, and recovery strategies. Through a shared perception-and-control interface, these skills guide subsequent execution in simulation and on real robots, with motions adapted to current observations and no updates to the underlying model weights. Extensive evaluations demonstrate that Real2Gym enables high-fidelity simulation environment reconstruction, outperforming GPT-6 Astra Direct Mode by 16.7% in success rate with approximately 74.9% fewer policy-execution tokens across these environments, while exceeding it by 33.3% in physical robot execution success rate across four tasks on a real Franka robot.
CommentsProject page: https://real2gym.github.io/
