从视频中进行手-物体交互预训练
Hand-Object Interaction Pretraining from Videos
AI总结：
本研究提出从视频的3D手物交互轨迹学习机器人操作先验的框架，通过生成式建模得到任务无关基础策略，微调后可高效适配下游任务，提升鲁棒性与泛化性。
摘要：
我们提出一种从3D手-物体交互轨迹中学习通用机器人操作先验的方法。我们构建了一个框架，利用真实场景视频生成感觉运动机器人轨迹。具体做法是将人手和被操作物体都映射到共享3D空间中，并将人类动作重定向为机器人动作。对这些数据进行生成式建模，我们得到了一个任务无关的基础策略，该策略捕捉了通用且灵活的操作先验。我们通过实验证明，使用强化学习（RL）和行为克隆（BC）对该策略进行微调，能够高效适配下游任务，同时相比现有方法提升了鲁棒性和泛化性。定性实验结果见：https://hgaurav2k.github.io/hop/
英文摘要：
We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimotor robot trajectories. We do so by lifting both the human hand and the manipulated object in a shared 3D space and retargeting human motions to robot actions. Generative modeling on this data gives us a task-agnostic base policy. This policy captures a general yet flexible manipulation prior. We empirically demonstrate that finetuning this policy, with both reinforcement learning (RL) and behavior cloning (BC), enables sample-efficient adaptation to downstream tasks and simultaneously improves robustness and generalizability compared to prior approaches. Qualitative experiments are available at: \url{https://hgaurav2k.github.io/hop/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
