可扩展的视觉-语言-动作模型预训练：利用真实人类活动视频实现机器人操作
Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos
AI总结：
本文提出利用无标注的人类手部活动视频，通过全自动分析生成对齐的VLA训练数据，预训练灵巧手部VLA模型，实现零样本能力和显著提升的微调性能，为可扩展的机器人具身智能奠定基础。
摘要：
本文提出了一种新颖的方法，利用大规模无脚本的真实人类手部活动视频语料库，对机器人操作的视觉-语言-动作（VLA）模型进行预训练。将人类手部视为灵巧的机器人末端执行器，我们证明了无需任何标注的“野外”自我中心人类视频，可以在任务粒度和标签方面，转换为与现有机器人V-L-A训练数据完全对齐的数据格式。这是通过开发一种全自动的、针对任意人类手部视频的整体性人类活动分析方法来实现的。该方法能够生成原子级别的手部活动片段及其语言描述，每个片段都附带逐帧的3D手部运动和相机运动。我们处理了大量自我中心视频，并创建了一个包含100万集和2600万帧的手部VLA训练数据集。该训练数据涵盖了广泛的对象和概念、灵巧操作任务以及现实生活中的环境变化，远超现有机器人数据的覆盖范围。我们设计了一种灵巧手部VLA模型架构，并在此数据集上对模型进行了预训练。该模型在完全未见过的真实世界观测中展现出强大的零样本能力。此外，在少量真实机器人动作数据上进行微调，显著提高了真实机器人实验中的任务成功率和对新物体的泛化能力。我们还展示了模型任务性能相对于预训练数据规模的令人瞩目的扩展行为。我们相信，这项工作为可扩展的VLA预训练奠定了坚实基础，推动机器人迈向真正通用的具身智能。
英文摘要：
This paper presents a novel approach for pretraining robotic manipulation Vision-Language-Action (VLA) models using a large corpus of unscripted real-life video recordings of human hand activities. Treating human hand as dexterous robot end-effector, we show that "in-the-wild" egocentric human videos without any annotations can be transformed into data formats fully aligned with existing robotic V-L-A training data in terms of task granularity and labels. This is achieved by the development of a fully-automated holistic human activity analysis approach for arbitrary human hand videos. This approach can generate atomic-level hand activity segments and their language descriptions, each accompanied with framewise 3D hand motion and camera motion. We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames. This training data covers a wide range of objects and concepts, dexterous manipulation tasks, and environment variations in real life, vastly exceeding the coverage of existing robot data. We design a dexterous hand VLA model architecture and pretrain the model on this dataset. The model exhibits strong zero-shot capabilities on completely unseen real-world observations. Additionally, fine-tuning it on a small amount of real robot action data significantly improves task success rates and generalization to novel objects in real robotic experiments. We also demonstrate the appealing scaling behavior of the model's task performance with respect to pretraining data scale. We believe this work lays a solid foundation for scalable VLA pretraining, advancing robots toward truly generalizable embodied intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
