EgoVLA：从第一人称人类视频学习视觉-语言-动作模型
EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos
AI总结：
本文提出EgoVLA，利用第一人称人类视频训练VLA模型，通过逆运动学和重定向将人类动作转为机器人动作，并用少量机器人演示微调，在Ego Humanoid Manipulation Benchmark上显著优于基线。
摘要：
用于模仿学习的真实机器人数据收集已推动机器人操作取得显著进展。然而，该过程中对机器人硬件的需求从根本上限制了数据规模。本文探索利用第一人称人类视频训练视觉-语言-动作（VLA）模型。使用人类视频的优势不仅在于其规模，更在于场景和任务的丰富性。通过在人类视频上训练预测人类手腕和手部动作的VLA，我们可以执行逆运动学和重定向，将人类动作转换为机器人动作。我们使用少量机器人操作演示对模型进行微调，得到机器人策略，即EgoVLA。我们提出一个名为Ego Humanoid Manipulation Benchmark的仿真基准，其中设计了多样的双手操作任务及演示。我们使用Ego Humanoid Manipulation Benchmark对EgoVLA进行微调和评估，结果显示其相比基线方法有显著提升，并通过消融实验验证了人类数据的重要性。视频可在我们的网站查看：https://rchalyang.github.io/EgoVLA
英文摘要：
Real robot data collection for imitation learning has led to significant advancements in robotic manipulation. However, the requirement for robot hardware in the process fundamentally constrains the scale of the data. In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos. The benefit of using human videos is not only for their scale but more importantly for the richness of scenes and tasks. With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and retargeting to convert the human actions to robot actions. We fine-tune the model using a few robot manipulation demonstrations to obtain the robot policy, namely EgoVLA. We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations. We fine-tune and evaluate EgoVLA with Ego Humanoid Manipulation Benchmark and show significant improvements over baselines and ablate the importance of human data. Videos can be found on our website: https://rchalyang.github.io/EgoVLA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
