H2R: 一种从视频中为机器人预训练的人到机器人的数据增强
H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
AI总结：
H2R通过将人类视角视频转换为机器人视角数据，弥合人机视觉差距，提升机器人预训练效果，实验显示在模拟和现实场景中均显著提升性能。
摘要：
大规模使用第一人称人类视频进行预训练已被证明对机器人学习有效。然而，由于人类手部与不同机器人手部之间存在显著的视觉差距，预训练模型可能对机器人学习不理想。为解决此问题，我们提出H2R，一种人类到机器人的数据增强管道，将第一人称人类视频转换为机器人视角的视觉数据。H2R估计人类手部姿态，将运动转换到模拟的机械臂上，通过分割和修复去除人类肢体，并将渲染的机器人化身与原始帧结合，使视觉差距在预训练过程中得到明确弥合。我们应用H2R来增强大规模第一人称人类视频数据集，如Ego4D和SSv2。为验证增强管道的有效性，我们引入基于CLIP的图像-文本相似性度量，定量评估机器人渲染帧与原始人类动作的语义保真度。我们通过在模拟和现实环境中的全面实验评估H2R。在模拟中，H2R在四个基准测试套件中持续提升下游成功率，获得1.3%-10.2%的提升。在现实实验中，H2R在UR5和双臂Franka/UR5操作平台上提升性能，实现3.3%-23.3%的成功率提升。我们进一步展示了H2R在跨化身泛化和与视觉-语言-动作模型的兼容性。这些结果表明，H2R通过减少人机领域之间的视觉差异，提高了机器人策略的泛化能力。
英文摘要：
Large-scale pre-training using egocentric human videos has proven effective for robot learning. However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots. To remedy this, we propose H2R, a human-to-robot data augmentation pipeline that converts egocentric human videos into robot-centric visual data. H2R estimates human hand pose from videos, retargets the motion to simulated robotic arms, removes human limbs via segmentation and inpainting, and composites rendered robot embodiments into the original frames with camera-aligned geometry. This process explicitly bridges the visual gap between human and robot embodiments during pre-training. We apply H2R to augment large-scale egocentric human video datasets such as Ego4D and SSv2. To verify the effectiveness of the augmentation pipeline, we introduce a CLIP-based image-text similarity metric that quantitatively evaluates the semantic fidelity of robot-rendered frames to the original human actions. We evaluate H2R through comprehensive experiments in both simulation and real-world settings. In simulation, H2R consistently improves downstream success rates across four benchmark suites-Robomimic, RLBench, PushT, and CortexBench-yielding gains of 1.3%-10.2% across different visual encoders and policy learning methods. In real-world experiments, H2R improves performance on UR5 and dual-arm Franka/UR5 manipulation platforms, achieving 3.3%-23.3% success rate gains across gripper-based, dexterous, and bimanual tasks. We further demonstrate the potential of H2R in cross-embodiment generalization and its compatibility with vision-language-action models. These results indicate that H2R improves the generalization ability of robotic policies by mitigating the visual discrepancies between human and robot domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
