MVP-LAM:通过跨视角重建学习以动作为中心的潜在动作
MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction
AI总结:
提出MVP-LAM模型,利用多视角视频通过跨视角重建目标学习与真实动作高度相关的潜在动作,提升动作预测和下游操作性能。
摘要:
从多样化人类视频中学习的潜在动作作为视觉-语言-动作(VLA)预训练的伪标签,但只有当它们对底层真实动作保持信息量时才能提供有效监督。为了有效监督,潜在动作应包含关于底层动作的信息,尽管这些信息不可直接获取。我们提出多视角潜在动作模型(MVP-LAM),该模型从多视角视频中学习与真实动作高度相关的潜在动作。MVP-LAM通过跨视角重建目标训练潜在动作,使得一个视角的潜在动作必须解释另一个视角的未来,从而减少对视角特定线索的依赖。在Bridge V2上,MVP-LAM生成更以动作为中心的潜在动作,与真实动作的互信息更高,动作预测性能提升,包括在分布外评估下。最后,使用MVP-LAM潜在动作预训练VLA模型提高了各种基准上的下游操作性能。代码和训练好的检查点可在https://jmsnu.github.io获取。
英文摘要:
Latent actions learned from diverse human videos serve as pseudo-labels for vision-language-action (VLA) pretraining, but provide effective supervision only if they remain informative about the underlying ground-truth actions. For effective supervision, latent actions should contain information about the underlying actions even though they are inaccessible. We propose Multi-ViewPoint Latent Action Moel (MVP-LAM), which learns latent actions that are highly informative about ground-truth actions from multi-view videos. MVP-LAM trains latent actions with a cross-viewpoint reconstruction objective, so that a latent action from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on various benchmarks. The code and trained checkpoints are available at https://jmsnu.github.io.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。