arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01067cs.RO
收藏

对大型行为模型在机器人操作中协同训练数据模态和策略的系统研究

A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation

Toyota Research Institute, Cambridge MA（丰田研究院）；Tsinghua University, Beijing, China（清华大学）

Fanqi Lin, Kushal Arora, Jean Mercat, Haruki Nishimura, Paarth Shah, Chen Xu, Mengchao Zhang, Mark Zolotas, Maya Angeles, Owen Pfannenstiehl, Andrew Beaulieu, Jose Barreiros

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究了机器人操作中协同训练不同数据模态和策略对行为模型性能的影响，发现视觉语言和跨身体数据显著提升泛化能力，而离散动作令牌无明显优势。

摘要：

大型行为模型通过将模仿学习扩展到大规模多任务机器人数据上的大尺度训练，展现了强大的灵巧操作能力，但其泛化能力仍受限于机器人数据覆盖不足。为了在不增加成本的额外数据收集的情况下扩展这一覆盖范围，近期的工作依赖于协同训练：从目标机器人数据和异质数据模态共同学习。然而，不同协同训练数据模态和策略如何影响策略性能仍缺乏深入理解。我们进行了一项大规模的实证研究，考察了五种协同训练数据模态：标准视觉语言数据、密集语言标注的机器人轨迹数据、跨身体机器人数据、人类视频以及离散机器人动作令牌，在单阶段和多阶段训练策略下。我们的研究利用了4000小时的机器人和人类操作数据以及5000万视觉语言样本来训练视觉语言动作策略。我们评估了89种策略，进行了58000次模拟回放和2835次现实世界回放。我们的结果表明，使用视觉语言和跨身体机器人数据的协同训练显著提高了对分布偏移、未见任务和语言跟随的泛化能力，而离散动作令牌变体则没有显著优势。结合有效的模态产生累积收益，并通过微调使机器人能够快速适应未见的长周期灵巧任务。仅训练机器人数据会降低视觉语言模型主干的视觉语言理解能力，而使用有效的模态进行协同训练可以恢复这些能力。显式地将动作生成条件化为从协同训练数据中学习到的推理轨迹并不在我们的模拟基准中提高性能。这些结果为构建可扩展的通用机器人策略提供了实用指导。

英文摘要：

Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalization remains limited by the insufficient robot data coverage. To expand this coverage without costly additional data collection, recent work relies on co-training: jointly learning from target robot data and heterogeneous data modalities. However, how different co-training data modalities and strategies affect policy performance remains poorly understood. We present a large-scale empirical study examining five co-training data modalities: standard vision-language data, dense language annotations for robot trajectories, cross-embodiment robot data, human videos, and discrete robot action tokens across single- and multi-phase training strategies. Our study leverages 4,000 hours of robot and human manipulation data and 50M vision-language samples to train vision-language-action policies. We evaluate 89 policies over 58,000 simulation rollouts and 2,835 real-world rollouts. Our results show that co-training with forms of vision-language and cross-embodiment robot data substantially improves generalization to distribution shifts, unseen tasks, and language following, while discrete action token variants yield no significant benefits. Combining effective modalities produces cumulative gains and enables rapid adaptation to unseen long-horizon dexterous tasks via fine-tuning. Training exclusively on robot data degrades the visiolinguistic understanding of the vision-language model backbone, while co-training with effective modalities restores these capabilities. Explicitly conditioning action generation on chain-of-thought traces learned from co-training data does not improve performance in our simulation benchmark. Together, these results provide practical guidance for building scalable generalist robot policies.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑