通过人类视频中的视觉-物理对齐实现空间感知的VLA预训练
Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
AI总结：
通过人类视频中的视觉-物理对齐实现空间感知的VLA预训练，提升机器人任务的稳健性和通用性。
摘要：
视觉-语言-动作（VLA）模型通过整合视觉感知与语言引导的策略学习，为机器人学习提供了一个有前景的范式。然而，大多数现有方法依赖于2D视觉输入在3D物理环境中执行动作，导致感知与动作定位之间存在显著差距。为弥合这一差距，我们提出了一个空间感知的VLA预训练范式，在预训练过程中显式地对齐视觉空间与物理空间，使模型在机器人策略学习之前获得3D空间理解。从预训练的视觉-语言模型出发，我们利用大规模的人类示范视频提取3D视觉和3D动作标注，形成一个新的监督源，将2D视觉观察与3D空间推理对齐。我们通过VIPA-VLA架构实例化这一范式，该架构包含一个3D视觉编码器，用于增强语义视觉表示的3D感知特征。当应用于下游机器人任务时，VIPA-VLA在2D视觉与3D动作之间实现了显著的定位提升，从而产生更加稳健和通用的机器人策略。
英文摘要：
Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining, enabling models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
