Video2Sim2Real：从单个人类视频实现全栈自主灵巧技能获取
Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video
AI总结：
提出Video2Sim2Real框架，从单个人类操作视频中重建数字孪生并提取运动先验，通过物体关键帧优化机器人配置，结合残差强化学习与碰撞感知规划，实现从仿真到真实世界的灵巧技能迁移。
摘要：
人类操作视频是机器人学习的便捷直观来源。然而，由于感知误差和具身差距，直接将人类灵巧性迁移到机器人仍然具有挑战性。为此，我们引入Video2Sim2Real，一个从单个人类操作视频中自主获取技能的全栈框架。我们的框架首先使用现成的基础模型重建适用于仿真器的数字孪生，并提取机器人和物体运动先验。与将提取的机器人运动视为整个执行过程中的可靠参考不同，我们的关键思想是恢复并利用从演示技能中获得的最基本监督来源：我们识别以物体为中心的关键帧，利用仿真器中的物体信息优化相应的机器人配置，并将这些配置作为锚点来细化机器人运动，使其最终对环境产生期望的影响。为了弥合剩余的仿真到现实差距，我们引入了一种仿真到现实策略，将对噪声和不完整感知的鲁棒性与手-物交互动力学的变化解耦。具体来说，我们通过模仿学习从噪声的真实世界点云中重新校准机器人配置，并利用残差强化学习进行局部手指级自适应，以确保鲁棒且有效的交互。最后，一个碰撞感知的运动规划模块实现了对新颖物体配置的空间泛化。在多个日常操作任务中，Video2Sim2Real在模拟任务成功率、安全性和轨迹一致性上优于众多基线，并且比现有技术实现了更好的仿真到现实迁移。这些结果展示了从人类视频自主获取灵巧技能的一条有前景的路径。
英文摘要：
Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
