DreamDojo：从大规模人类视频中学习通用机器人世界模型
DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos
AI总结：
DreamDojo通过大规模人类视频学习通用机器人世界模型，解决动作标签稀缺问题，实现高精度物理理解和实时交互。
摘要：
能够模拟不同环境中动作结果将革新大规模通用智能体的发展。然而，建模这些世界动态，尤其是对于灵巧机器人任务，由于数据覆盖有限和动作标签稀缺而面临重大挑战。为此，我们引入了DreamDojo，一个基础世界模型，从44000小时的目击人类视频中学习多样化的交互和灵巧控制。我们的数据混合代表了到目前为止最大的视频数据集，涵盖了广泛的一日生活场景，具有多样化的物体和技能。为了解决动作标签稀缺的问题，我们引入了连续潜在动作作为统一的代理动作，增强从无标签视频中转移交互知识。在小规模目标机器人数据上进行微调后，DreamDojo表现出对物理的深刻理解和精确的动作可控性。我们还设计了一条蒸馏流水线，使DreamDojo加速到每秒10.81帧的实时速度，并进一步提高上下文一致性。我们的工作基于生成世界模型实现了几个重要的应用，包括实时远程操作、策略评估和基于模型的规划。在多个具有挑战性的分布外（OOD）基准上的系统评估验证了我们的方法在模拟开放世界、接触丰富任务中的重要性，为通用机器人世界模型铺平了道路。
英文摘要：
Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
