Being-H0：基于大规模人类视频的视觉-语言-动作预训练
Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
AI总结：
Being-H0通过物理指令调优和部件级运动分词，利用大规模人类视频预训练视觉-语言-动作模型，以解决灵巧操作的数据瓶颈并提升真实机器人操作性能。
摘要：
我们提出Being-H0，一个在大规模人类视频上训练的灵巧视觉-语言-动作模型（VLA）。现有VLA在需要高灵巧性的复杂操作任务上表现不佳，并且对新场景和新任务的泛化能力较差，主要原因是它们依赖存在显著仿真到现实差距的合成数据，或依赖缺乏规模和多样性的遥操作演示。为解决这一数据瓶颈，我们提出将人手作为基础操作器，利用网络数据中丰富的灵巧性和可扩展性。我们的方法核心是物理指令调优，这是一种新颖的训练范式，结合了来自人类视频的大规模VLA预训练、用于3D推理的物理空间对齐，以及面向机器人任务的后训练适配。此外，我们引入了一种部件级运动分词方法，该方法实现了毫米级重建精度，从而为动作学习建模精确的手部轨迹。为支持所提出的范式，我们进一步开发了一套全面的数据整理流程，将异构来源——包括动作捕捉、VR和纯RGB视频——整合为一个包含数百万条基于运动的指令实例的大规模数据集。我们通过实验证明了Being-H0在手部运动生成和指令跟随方面的卓越性能，并且它随模型和数据规模扩展良好。重要的是，我们观察到在应用物理指令调优后，Being-H0在真实世界机器人操作中取得了预期的收益。更多详情见https://beingbeyond.github.io/Being-H0。
英文摘要：
We introduce Being-H0, a dexterous Vision-Language-Action model (VLA) trained on large-scale human videos. Existing VLAs struggle with complex manipulation tasks requiring high dexterity and generalize poorly to novel scenarios and tasks, primarily due to their reliance on synthetic data with significant sim-to-real gaps or teleoperated demonstrations lacking scale and diversity. To address this data bottleneck, we propose leveraging human hands as a foundation manipulator, capitalizing on the rich dexterity and scalability present in web data. Our approach centers on physical instruction tuning, a novel training paradigm that combines large-scale VLA pretraining from human videos, physical space alignment for 3D reasoning, and post-training adaptation for robotic tasks. Additionally, we introduce a part-level motion tokenization method which achieves millimeter-level reconstruction accuracy to model precise hand trajectories for action learning. To support our proposed paradigm, we further develop a comprehensive data curation pipeline that integrates heterogeneous sources -- including motion capture, VR, and RGB-only videos -- into a large-scale dataset with millions of motion-based instructional instances. We empirically show the excellence of Being-H0 in hand motion generation and instruction following, and it also scales well with model and data sizes. Importantly, we observe the expected gains of Being-H0 in real-world robotic manipulation as physical instruction tuning is applied. More details are available at https://beingbeyond.github.io/Being-H0.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
