arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.18822cs.RO

EgoMatrix：来自第一人称视角人类视频的物理保真灵巧机器人演示

EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos

Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）

Ziheng Wang, Yanming Shao, Ting Mao, Kaihan Chen, Yiqun Wang, Xuanye Wu, Yao Mu

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出EgoMatrix端到端数据引擎，将第一人称人类视频转为可直接在真实机器人重放的双臂灵巧轨迹与配对观测，在仿真中实现54.84%端到端成功率，优于基线及现有方法。

摘要：

第一人称视角人类视频为收集灵巧操作数据提供了一种可扩展的替代方案，可用于机器人遥操作。然而，两个挑战限制了它们作为可靠训练数据的使用：从单目RGB图像进行准确的手-物体交互（HOI）重建，以及高质量的机器人演示，尤其是视觉观测。我们提出了EgoMatrix，这是一种端到端数据引擎，可将单目第一人称视角人类视频转换为双臂灵巧轨迹和配对的机器人观测视频，这些轨迹可直接在真实机器人上重放。它重建HOI时侧重相对几何和接触，而非精确的度量场景恢复，随后使用重定向技术和强化学习（RL）在仿真中复现演示的任务。我们从这些仿真轨迹中联合渲染机器人手和物体，以生成与状态和控制记录同步的物理保真观测。在多样的、与任务兼容的背景中渲染相同轨迹，可在保留机器人-物体运动和接触的同时扩展配对训练数据。在TACO、OakInk2和自行收集的数据上，EgoMatrix在仿真中实现了54.84%的端到端成功率，而最强基线的成功率为14.52%。在两个公开基准的大多数指标上，其重建达到了最先进的性能。定性结果显示，与现有方法相比，它呈现出更保真的手-物体交互，而特征空间分析表明，它与真实机器人观测的对齐度比GPT-6更高。项目页面可在提供的链接获取。

英文摘要：

Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra.Our project page is available at https://spikew726.github.io/EgoMatrix/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑