arXiv 2606.221362026-06-24cs.RO

Wh0: 生成式世界模型作为自我中心人类手部操作数据的可扩展来源

Wh0: Generative World Models as Scalable Sources of Egocentric Human Hand Manipulation Data

Shanghai Innovation Institute（上海创新研究院）；Nanjing University（南京大学）；Shanghai Jiaotong University（上海交通大学）

Yangtao Chen, Zixuan Chen, Peiyang Wang, Yong-Lu Li, Jing Huo, Jieqi Shi, Yang Gao

备注：: Under review. The first three authors contributed equally to this work

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出Wh0框架，利用生成式视频世界模型产生自我中心人-物交互数据集WM-H，通过手部运动重建和视觉编辑转化为机器人可训练监督，提升预训练灵巧VLA模型在真实任务上的零样本成功率。

摘要：

扩展灵巧操作需要跨物体、场景和任务的泛化能力，但现有数据源在规模与场景/具身对齐之间存在权衡：遥操作数据与机器人部署对齐良好但收集成本高；仿真可扩展但受限于仿真到真实的差距；真实自我中心视频有效扩展但与机器人部署仍不对齐。我们提出Wh0，一个利用生成式视频世界模型作为自我中心人类手部操作数据的可扩展和可控来源的框架，以解锁预训练灵巧VLA模型的操作能力。Wh0以语言、物体和场景为条件，使用生成式世界模型生成WM-H，一个包含50k个自我中心人-物交互视频片段的数据集。然后，Wh0通过手部运动重建和视觉编辑将生成的视频转换为机器人可训练的监督信号。与有限量的真实机器人数据共同训练后，WM-H使预训练的VLA模型适应灵巧操作部署。在18个真实世界灵巧操作任务中，与仅在机器人数据上后训练的模型相比，Wh0将未见任务上的零样本成功率从8.3%提高到38.9%。消融研究进一步表明，可扩展生成和场景/具身对齐是性能提升的关键驱动因素。视频和开源代码可在我们的项目网站上找到：此 https URL。

英文摘要：

Scaling dexterous manipulation requires generalization across objects, scenes, and tasks, yet existing data sources face a trade-off between scale and scene/embodiment alignment: teleoperation data is well aligned with robot deployment but expensive to collect; simulation is scalable but limited by the sim-to-real gap; and real egocentric videos scale effectively but remain misaligned with robot deployment. We propose Wh0, a framework that uses generative video world models as scalable and controllable sources of egocentric human-hand manipulation data to unlock the manipulation capabilities of pretrained dexterous VLA models. Conditioned on language, objects, and scenes, Wh0 uses a generative world model to produce WM-H, a 50k-episode dataset of egocentric human-object interaction videos. Wh0 then converts the generated videos into robot-trainable supervision through hand motion reconstruction and visual editing. Co-trained with a limited amount of real robot data, WM-H adapts pretrained VLA models to dexterous manipulation deployment. Across 18 real-world dexterous manipulation tasks, compared with a model post-trained only on robot data, Wh0 improves zero-shot success on unseen tasks from 8.3% to 38.9%. Ablation studies further show that scalable generation and scene/embodiment alignment are key drivers of performance gains. Videos and open-source code can be found on our project website: https://chenyt31.github.io/wh0.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。