arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.16587cs.RO
收藏

Human2Robot:从配对的人机视频中学习机器人动作

Human2Robot: Learning Robot Actions from Paired Human-Robot Videos

Sicheng Xie, Haidong Cao, Zejia Weng, Zhen Xing, Haoran Chen, Shiwei Shen, Jiaqi Leng, Zuxuan Wu, Yu-Gang Jiang

AI总结:

针对现有方法依赖粗略对齐视频对、忽略细粒度帧级动态的问题,提出将人机细粒度对齐视为条件视频生成问题的范式,构建含2600段精确同步片段的H&R数据集,设计Human2Robot框架,实现已见任务高性能与新场景单样本泛化。

摘要:

从人类演示中提取知识是机器人学习和行动的一种有前景的方法。现有方法通常依赖粗略对齐的视频对,因此局限于学习全局或任务级特征,往往忽略复杂操作和泛化到新任务所需的细粒度帧级动态。我们认为这一局限源于数据集不足及其启发的方法形成的恶性循环。为打破该循环,我们提出一种范式转变,将细粒度人机对齐视为条件视频生成问题。为此,我们首先引入H&R数据集——一个新的第三人称数据集,包含2600段通过VR遥操作系统采集的精确同步的人机运动片段。然后我们提出Human2Robot框架,利用该数据:它采用视频预测模型,通过从人类输入生成机器人视频来学习机器人动态的丰富隐式表示,进而指导解耦的动作解码器。我们的真实世界实验表明,该方法不仅在已见任务上实现高性能,还在新位置、物体、实例甚至新任务类别上表现出显著的单样本泛化能力。

英文摘要:

Distilling knowledge from human demonstrations is a promising way for robots to learn and act. Existing methods, which often rely on coarsely-aligned video pairs, are typically constrained to learning global or task-level features. As a result, they tend to neglect the fine-grained frame-level dynamics required for complex manipulation and generalization to novel tasks. We posit that this limitation stems from a vicious circle of inadequate datasets and the methods they inspire. To break this cycle, we propose a paradigm shift that treats fine-grained human-robot alignment as a conditional video generation problem. To this end, we first introduce H&R, a novel third-person dataset containing 2,600 episodes of precisely synchronized human and robot motions, collected using a VR teleoperation system. We then present Human2Robot, a framework designed to leverage this data. Human2Robot employs a Video Prediction Model to learn a rich and implicit representation of robot dynamics by generating robot videos from human input, which in turn guides a decoupled action decoder. Our real-world experiments demonstrate that this approach not only achieves high performance on seen tasks but also exhibits significant one-shot generalization to novel positions, objects, instances, and even new task categories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑