arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.13732cs.CVcs.AIcs.RO

LAPT-Scale:基于人类视频的潜在动作预训练扩展具身策略

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Joy Future Academy, JD(京东探索研究院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Tsinghua SIGS, Tsinghua University(清华大学深圳国际研究生院);Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学院)

Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng

AI总结:

本文提出LAPT-Scale范式,利用人类视频的隐式动作监督训练机器人策略,使真实机器人任务成功率从63.3%提升至85.0%,且策略可泛化到新视觉条件。

摘要:

通用机器人策略的扩展受限于具身交互数据的采集成本过高,大规模第一人称视角人类视频是丰富的替代方案,但这类视频缺乏机器人动作标签,且现有动作标注方法在现实世界视觉复杂性和多样操作动态下往往存在噪声。本文提出LAPT-Scale,这是一种可扩展的潜在动作预训练范式,包含两个组件:LAPT-Scale-LAM通过解释人类视频中帧间视觉过渡学习连续潜在动作,并将其作为人类衍生的监督信号;LAPT-Scale-Policy通过人类视频潜在动作预训练、目标机器人具身对齐以及特定任务的后训练来训练VLA策略。与先前的向量量化潜在动作模型(LAMs)不同,LAPT-Scale-LAM采用连续潜在空间以保留细粒度的运动和交互动态。我们进一步结合感知、语义和光流对齐信号,以改进超越像素重建的过渡级监督。在23.3K小时的第一人称视角视频上训练后,LAPT-Scale-LAM为策略预训练提供隐式动作监督,产生一致的扩展增益,当标注人类视频规模从1K小时增加到5K小时时,真实机器人的成功率从63.3%提升至85.0%。所学策略还能泛化到未见过的视觉条件。

英文摘要:

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑