释放大规模视频生成式预训练在视觉机器人操作中的潜力
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
AI总结：
本文提出GR-1，一个GPT风格的统一Transformer模型，通过大规模视频生成式预训练，在多任务视觉机器人操作中显著提升成功率，在CALVIN基准和真实机器人上均优于基线方法。
摘要：
生成式预训练模型通过在语言和视觉领域学习有用的表示，已展现出显著的有效性。在本文中，我们扩展了这一有效性的范围，表明视觉机器人操作可以显著受益于大规模视频生成式预训练。我们引入了GR-1，一个直接的GPT风格模型，专为多任务语言条件视觉机器人操作而设计。GR-1以语言指令、观测图像序列和机器人状态序列作为输入，以端到端的方式预测机器人动作以及未来图像。得益于灵活的设计，GR-1在大型视频数据集上预训练后，可以无缝地在机器人数据上进行微调。我们在具有挑战性的CALVIN基准测试和真实机器人上进行了大量实验。在CALVIN基准上，我们的方法优于最先进的基线方法，将成功率从88.9%提高到94.9%。在零样本未见场景泛化设置中，GR-1将成功率从53.3%提高到85.4%。在真实机器人实验中，GR-1也优于基线方法，并在泛化到未见场景和物体方面展现出强大潜力。我们提供了初步证据，表明一个统一的GPT风格Transformer，辅以大规模视频生成式预训练，对多任务视觉机器人操作展现出显著的泛化能力。项目页面：https://GR1-Manipulation.github.io
英文摘要：
Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a straightforward GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
