运动聚焦的潜在动作使跨实体VLA训练能从人类自我中心视频中学习
Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos
- Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
- Tianfu Jiangxi Laboratory（天府江西实验室）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出基于潜在动作的框架，利用混合解耦VQ-VAE从无标签人类视频中提取通用动作先验，通过意图-感知解耦策略减少动作幻觉，仅需50条轨迹即可适配下游任务。
AI中文摘要：
训练通用视觉-语言-动作（VLA）模型通常需要大量、多样化的机器人数据集，并带有高保真动作标注。尽管自我中心的人类操作视频丰富且捕捉了显著的环境多样性，但缺乏动作标签使其难以在传统训练范式下使用。为解决这一问题，我们提出了一种基于潜在动作的框架，旨在从无标签人类视频中提取通用动作先验。该架构采用混合解耦VQ-VAE，通过物理掩码将运动动态与环境背景解耦，从而构建跨实体动作码本。通过在人类视频上使用码本进行预训练，VLM骨干网络学习到动作意图的深层表示。为了适应特定实体，我们引入了一种意图-感知解耦策略，其中VLM预测动作意图，而一个独立的冻结视觉编码器为动作专家提供状态特定特征，从而减少动作幻觉。在仿真和真实环境中的结果表明，我们的方法仅在无标签人类视频上预训练，与在大量标注数据集上训练的最先进VLA模型相比具有竞争力，且仅需50条轨迹进行下游适配。
英文摘要：
Training generalist Vision-Language-Action(VLA) models typically requires massive, diverse robotic datasets with high-fidelity action annotations. While egocentric human manipulation videos are abundant and capture significant environmental diversity, the absence of action labels makes them difficult to use in conventional training paradigms. To address this, we propose a latent-action-based framework designed to extract general action priors from unlabeled human videos. The architecture features a Hybrid Disentangled VQ-VAE that decouples motion dynamics from environmental backgrounds through physical masks, enabling the construction of a cross-embodiment action codebook. By pre-training on human videos with the codebook, the VLM backbone learns deep representations of action intent. For adaptation to specific embodiments, we introduce an intent-perception decoupling strategy where the VLM predicts the action intent while a separate frozen visual encoder provides state-specific features to the action expert, thereby reducing action hallucinations. Results in simulation and real-world environments show that our method, pre-trained exclusively on unlabeled human videos, performs competitively with state-of-the-art VLA models trained on massive annotated datasets, requiring only 50 trajectories for downstream adaptation.
