arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.41877cs.ROcs.CV
收藏

Movens-Zero:基于大规模人类视频扩展人形运动基础模型

Movens-Zero: Scaling Humanoid Motion Foundation Models with Large-Scale Human Videos

Zhejiang University(浙江大学);Unitree Robotics(宇树机器人);Joint Laboratory of Embodied Intelligence, ZJU & Unitree Robotics(浙江大学与宇树机器人联合具身智能实验室);Bit Oasis;HiThink Research(HiThink研究院);Microsoft Research(微软研究院);East China Normal University(华东师范大学);Peking University(北京大学);National University of Singapore(新加坡国立大学)
Wendong Bu, Tinghong Ye, Qizhou Wang, Jiacheng Li, Zhiqi Ge, Yuze Lin, Zhongqi Yue, Yi Su, Jingyu Wang, Kai Shen, Jingdong Liu, Chengyuan Deng, Xiaoxuan Chen, X… 展开作者

Wendong Bu, Tinghong Ye, Qizhou Wang, Jiacheng Li, Zhiqi Ge, Yuze Lin, Zhongqi Yue, Yi Su, Jingyu Wang, Kai Shen, Jingdong Liu, Chengyuan Deng, Xiaoxuan Chen, Xin Tan, Jiaming Ji, Wenqiao Zhang, Chenghong Jiang, Tuoyu Li, Siliang Tang, Jun Xiao, Tat-Seng Chua, Yueting Zhuang, Xingxing Wang, Juncheng Li

AI总结:

该研究构建了大规模人形运动数据集Movens-5M,提出可扩展的Movens-Zero架构,结合任务混合流匹配训练与多模态实时分块技术,在四轨基准Movens-Intent上验证了数据与模型规模扩展对运动生成及执行的提升效果。

摘要:

扩展人形运动基础模型受限于多样化、可由机器人执行的数据稀缺,以及从异构多模态输入中联合学习并支持实时闭环控制的难度。我们提出Movens-5M,其通过片段生成、质量过滤、语义标注、运动恢复,以及人形重定向与执行验证的物理优化,从原始互联网视频构建而成。该数据集包含1500万个人类运动序列和500万个可由机器人执行的人形运动序列,时长超10000小时,与文本、RGB视频、语音和音乐对齐。基于此数据,我们引入Movens-Zero,这是一种可扩展的理解-生成-执行架构,支持跨模态联合学习与实时闭环执行。任务混合流匹配训练在7项生成任务中联合优化单个动作专家,实现从文本、语音、音乐和视频直接生成运动。多模态实时分块在持续更新的指令与机器人状态下连接重叠分块,以实现连续闭环执行。我们进一步提出Movens-Intent,这是一个四轨基准,包含8192个经专家验证的条件-运动对,覆盖文本、语音、音乐和视频。实验显示,当将执行验证数据从1%扩展至100%、动作专家容量从6400万参数扩展至9.11亿参数时,所有四个模态均取得持续提升,同时证明数据质量与跨模态监督可改善生成与执行效果。

英文摘要:

Scaling humanoid motion foundation models is constrained by the scarcity of diverse, robot-executable data and by the difficulty of jointly learning from heterogeneous multimodal inputs while supporting real-time closed-loop control. We present Movens-5M, constructed from raw Internet videos through clip generation, quality filtering, semantic annotation, motion recovery, and physical refinement with humanoid retargeting and execution verification. It contains 15 million human-motion sequences and 5 million physically executable humanoid-motion sequences spanning over 10,000 hours, aligned with text, RGB video, speech, and music. Building on this data, we introduce Movens-Zero, a scalable understanding-generation-execution architecture that supports joint learning across modalities and real-time closed-loop execution. Task-mixed flow-matching training jointly optimizes a single action expert across seven generation tasks, enabling direct motion generation from text, speech, music, and video. Multimodal Real-Time Chunking connects overlapping chunks under continuously updated instructions and robot states for continuous closed-loop execution. We further introduce Movens-Intent, a four-track benchmark with 8,192 expert-verified condition-motion pairs across text, speech, music, and video. Experiments show consistent gains across all four modalities when scaling execution-verified data from 1% to 100% and action-expert capacity from 64M to 911M parameters, while demonstrating that data quality and cross-modal supervision improve generation and execution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑