MimicDroid：基于人类游戏视频的上下文学习用于人形机器人操作
MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos
AI总结：
提出MimicDroid框架，利用人类游戏视频作为唯一训练数据，通过上下文学习使人形机器人高效适应新操作任务，在仿真与真实世界中均显著优于现有方法。
摘要：
我们旨在使人形机器人能够从少量视频示例中高效解决新的操作任务。上下文学习（ICL）因其测试时的数据效率和快速适应性，成为实现这一目标的有前景的框架。然而，当前的ICL方法依赖劳动密集型的遥操作数据进行训练，限制了可扩展性。我们提出使用人类游戏视频——人们与环境自由交互的连续、无标签视频——作为可扩展且多样的训练数据源。我们引入了MimicDroid，它使类人机器人能够仅使用人类游戏视频作为训练数据来执行ICL。MimicDroid提取具有相似操作行为的轨迹对，并训练策略以在给定一条轨迹的条件下预测另一条轨迹的动作。通过这一过程，模型获得了在测试时适应新对象和环境的ICL能力。为弥合具身差距，MimicDroid首先将RGB视频估计的人体手腕姿态重定向至人形机器人，利用运动学相似性。训练中还应用随机块掩码以减少对人类特定线索的过拟合，增强对视觉差异的鲁棒性。为评估人形机器人的少样本学习，我们引入了一个开源仿真基准，包含递增的泛化难度。MimicDroid优于现有方法，在真实世界中实现近两倍的成功率。更多资料见：ut-austin-rpl.github.io/MimicDroid
英文摘要：
We aim to enable humanoid robots to efficiently solve new manipulation tasks from a few video examples. In-context learning (ICL) is a promising framework for achieving this goal due to its test-time data efficiency and rapid adaptability. However, current ICL methods rely on labor-intensive teleoperated data for training, which restricts scalability. We propose using human play videos -- continuous, unlabeled videos of people interacting freely with their environment -- as a scalable and diverse training data source. We introduce MimicDroid, which enables humanoids to perform ICL using human play videos as the only training data. MimicDroid extracts trajectory pairs with similar manipulation behaviors and trains the policy to predict the actions of one trajectory conditioned on the other. Through this process, the model acquired ICL capabilities for adapting to novel objects and environments at test time. To bridge the embodiment gap, MimicDroid first retargets human wrist poses estimated from RGB videos to the humanoid, leveraging kinematic similarity. It also applies random patch masking during training to reduce overfitting to human-specific cues and improve robustness to visual differences. To evaluate few-shot learning for humanoids, we introduce an open-source simulation benchmark with increasing levels of generalization difficulty. MimicDroid outperformed state-of-the-art methods and achieved nearly twofold higher success rates in the real world. Additional materials can be found on: ut-austin-rpl.github.io/MimicDroid
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
