Moto：潜在运动标记作为从视频学习机器人操作的桥接语言
Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos
AI总结：
本文提出Moto，利用潜在运动标记作为桥接语言，通过自回归预训练从视频中学习运动知识，并协同微调迁移至机器人操作，显著提升基准性能。
摘要：
近年来，在广泛语料库上预训练的大型语言模型在各类自然语言处理任务中取得了显著成功，且仅需极少的微调。这一成功为长期受限于动作标注数据高成本的机器人领域带来了新的希望。我们提出疑问：鉴于包含交互相关知识的丰富视频数据可作为庞大的“语料库”，类似的生成式预训练方法能否有效应用于增强机器人学习？关键挑战在于找到一种有效的表示用于自回归预训练，以有益于机器人操作任务。受人类通过观察动态环境学习新技能的启发，我们提出有效的机器人学习应强调与运动相关的知识，这类知识与低级动作紧密相连且与硬件无关，便于将学到的运动迁移到实际机器人动作中。为此，我们引入了Moto，它通过潜在运动标记器将视频内容转换为潜在运动标记序列，以无监督方式从视频中学习运动的桥接“语言”。我们通过运动标记自回归预训练Moto-GPT，使其能够捕获多样的视觉运动知识。预训练后，Moto-GPT展现出生成语义可解释的运动标记、预测合理的运动轨迹以及通过输出似然评估轨迹合理性的能力。为将学到的运动先验迁移到真实机器人动作，我们实施了一种协同微调策略，无缝衔接潜在运动标记预测与真实机器人控制。大量实验表明，微调后的Moto-GPT在机器人操作基准上展现出卓越的鲁棒性和效率，突显了其将视频数据知识迁移到下游视觉操作任务的有效性。
英文摘要：
Recent developments in Large Language Models pre-trained on extensive corpora have shown significant success in various natural language processing tasks with minimal fine-tuning. This success offers new promise for robotics, which has long been constrained by the high cost of action-labeled data. We ask: given the abundant video data containing interaction-related knowledge available as a rich "corpus", can a similar generative pre-training approach be effectively applied to enhance robot learning? The key challenge is to identify an effective representation for autoregressive pre-training that benefits robot manipulation tasks. Inspired by the way humans learn new skills through observing dynamic environments, we propose that effective robotic learning should emphasize motion-related knowledge, which is closely tied to low-level actions and is hardware-agnostic, facilitating the transfer of learned motions to actual robot actions. To this end, we introduce Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer, learning a bridging "language" of motion from videos in an unsupervised manner. We pre-train Moto-GPT through motion token autoregression, enabling it to capture diverse visual motion knowledge. After pre-training, Moto-GPT demonstrates the promising ability to produce semantically interpretable motion tokens, predict plausible motion trajectories, and assess trajectory rationality through output likelihood. To transfer learned motion priors to real robot actions, we implement a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control. Extensive experiments show that the fine-tuned Moto-GPT exhibits superior robustness and efficiency on robot manipulation benchmarks, underscoring its effectiveness in transferring knowledge from video data to downstream visual manipulation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
