arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.16283cs.ROcs.CVcs.LGeess.IV
收藏

Gen2Act:在新型场景中生成人类视频以实现可泛化的机器人操作

Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation

Google DeepMind(谷歌DeepMind);The Robotics Institute, Carnegie Mellon University(卡内基梅隆大学机器人研究所);Computer Science Department, Stanford University(斯坦福大学计算机科学系)

Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, Sean Kirmani

AI总结:

本文提出 Gen2Act 方法,通过预训练网络视频生成模型预测运动信息,并以此条件化单一机器人策略,在仅用少量机器人数据的情况下实现了对未见物体和新动作的泛化操作。

摘要:

机器人操作策略如何泛化到涉及未见物体类型和新动作的新任务?本文提供了一种解决方案:通过人类视频生成从网络数据中预测运动信息,并利用生成的视频作为机器人策略的条件。我们没有尝试扩展昂贵的机器人数据收集,而是展示了如何利用在易获取的网络数据上训练的视频生成模型来实现泛化。我们的方法 Gen2Act 将语言条件操作转化为零样本人类视频生成,随后由一个基于生成视频进行条件控制的单一策略来执行。在训练策略时,相比视频预测模型的训练数据,我们使用的机器人交互数据少了一个数量级。Gen2Act 完全不需要对视频模型进行微调,我们直接使用预训练模型来生成人类视频。我们在多样化的真实世界场景中的结果表明,Gen2Act 能够操作未见物体类型,并执行机器人数据中不存在的任务的新动作。

英文摘要:

How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑