CLAP: 从人类视频中学习视觉-语言-动作模型的对比潜在动作预训练
CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
AI总结：
提出CLAP框架，通过对比学习将人类视频与机器人动作词汇对齐，利用伪标签训练VLA模型，实现从人类视频到机器人执行的有效技能迁移。
摘要：
通用视觉-语言-动作模型仍然受限于机器人数据的稀缺性，而人类视频演示则相对丰富。现有的潜在动作模型试图利用视频数据，但常常遭受视觉纠缠，编码噪声而非操作技能。为了解决这一限制，我们提出了对比潜在动作预训练（CLAP），该框架首先使用Act-VAE从机器人轨迹中学习可执行的动作标记词汇，然后通过对比学习将人类视觉转换与该词汇对齐。这种对齐将未标记的人类视频映射到物理上可行的潜在动作空间，而不是重建外观。基于对齐的标记，我们使用机器人演示和伪标记的人类视频训练CLAP-NTP作为自回归VLA，保持指令遵循和物体泛化能力。为了部署和目标域适应，我们进一步引入了一种后训练策略，该策略将CLAP-RF（一种用于低延迟连续动作块预测的整流流动作头）与知识匹配正则化相结合，以在微调期间保留预训练的语义知识。大量实验表明，CLAP在竞争基线上取得了强劲的性能，同时实现了从人类视频到机器人执行的有效技能迁移。
英文摘要：
Generalist Vision-Language-Action models remain constrained by the scarcity of robotic data relative to the abundance of human video demonstrations. Existing Latent Action Models attempt to use video data but often suffer from visual entanglement, encoding noise rather than manipulation skills. To address this limitation, we propose Contrastive Latent Action Pretraining (CLAP), a framework that first uses Act-VAE to learn an executable action-token vocabulary from robot trajectories and then aligns human visual transitions with this vocabulary through contrastive learning. This alignment maps unlabeled human videos into a physically grounded latent action space rather than reconstructing appearance. Building on the aligned tokens, we train CLAP-NTP as an autoregressive VLA using robot demonstrations and pseudo-labeled human videos, preserving instruction following and object generalization. For deployment and target-domain adaptation, we further introduce a post-training strategy that combines CLAP-RF, a Rectified Flow action head for low-latency continuous action chunk prediction, with Knowledge Matching regularization to preserve pretrained semantic knowledge during fine-tuning. Extensive experiments show that CLAP achieves strong performance against competitive baselines while enabling effective skill transfer from human videos to robotic execution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
