HERMES：面向移动灵巧操作的多源运动数据人机具身学习
HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation
AI总结：
针对多源人手运动向机器人行为转化难、策略泛化性不足等问题，提出HERMES框架，通过统一强化学习、深度图像端到端sim2real迁移及增强PnP定位的导航模型，实现移动双臂灵巧操作，在多样真实场景中表现出良好泛化性。
摘要：
利用人类运动数据赋予机器人多样化操作技能已成为机器人操作领域颇具前景的研究范式。然而，将多源人类手部运动转化为可行的机器人行为仍存在挑战，尤其是对于配备多指灵巧手、具有复杂高维动作空间的机器人而言。此外，现有方法往往难以生成能够适应多样环境条件的策略。本文提出了HERMES，一种面向移动双臂灵巧操作的人机学习框架。首先，HERMES构建了一种统一的强化学习方法，可将来自多源的异构人类手部运动无缝转化为物理上合理的机器人行为。随后，为缓解sim2real（仿真到现实）差距，我们设计了一种基于深度图像的端到端sim2real迁移方法，以提升对真实场景的泛化能力。此外，为实现机器人在多样非结构化环境中的自主运行，我们为导航基础模型增强了闭环Perspective-n-Point（PnP，透视n点）定位机制，确保视觉目标的精确对齐，有效衔接自主导航与灵巧操作。大量实验结果表明，HERMES在多样的真实场景中始终表现出可泛化的行为，成功完成了众多复杂的移动双臂灵巧操作任务。项目页面：https://gemcollector.github.io/HERMES/。
英文摘要：
Leveraging human motion data to impart robots with versatile manipulation skills has emerged as a promising paradigm in robotic manipulation. Nevertheless, translating multi-source human hand motions into feasible robot behaviors remains challenging, particularly for robots equipped with multi-fingered dexterous hands characterized by complex, high-dimensional action spaces. Moreover, existing approaches often struggle to produce policies capable of adapting to diverse environmental conditions. In this paper, we introduce HERMES, a human-to-robot learning framework for mobile bimanual dexterous manipulation. First, HERMES formulates a unified reinforcement learning approach capable of seamlessly transforming heterogeneous human hand motions from multiple sources into physically plausible robotic behaviors. Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth image-based sim2real transfer method for improved generalization to real-world scenarios. Furthermore, to enable autonomous operation in varied and unstructured environments, we augment the navigation foundation model with a closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise alignment of visual goals and effectively bridging autonomous navigation and dexterous manipulation. Extensive experimental results demonstrate that HERMES consistently exhibits generalizable behaviors across diverse, in-the-wild scenarios, successfully performing numerous complex mobile bimanual dexterous manipulation tasks. Project Page:https://gemcollector.github.io/HERMES/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
