arXiv 2606.080572026-06-06cs.ROcs.AI

EgoAERO：无需物体资产，从单个第一人称视频学习灵巧操作

EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

School of Astronautics, Harbin Institute of Technology（哈尔滨工业大学航天学院）；Lumos Robotic；Suzhou Research Institute, Harbin Institute of Technology（哈尔滨工业大学苏州研究院）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；Nanjing University（南京大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Fudan University（复旦大学）

Yichen Niu, Haoran Lv, Xinrui Zhang, Xueyao Wan, Shiyu Gao, Ying Ai, Hui Xu, Yongqi Hu, Hengyi Zhang, Yang Xie, Zhaxizhuoma, Yue Zhao, Zhenshan Bing, Yan Ding, Jianxing Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出EgoAERO框架，无需物体资产，从单个第一人称RGB-D视频中通过无资产物体跟踪与重建、自我运动补偿和自适应接触优化重建接触一致的手-物轨迹，并利用两阶段残差学习转化为机器人策略，实现单次演示的灵巧操作。

摘要：

第一人称RGB-D视频提供了人类灵巧操作演示的自然来源，但现有数据难以用于机器人学习，因为物体姿态、几何和接触信息常常缺失或需要预先扫描的物体资产。我们提出EgoAERO，这是第一个无需物体资产、从单个第一人称RGB-D人类演示中学习灵巧操作的框架。EgoAERO通过无资产物体跟踪与重建、自我运动补偿和自适应接触优化重建接触一致的手-物轨迹，然后利用两阶段残差学习将其转化为机器人策略。我们进一步引入在线质量评估机制，并构建EgoDex-R，一个包含430万RGB-D帧的大规模第一人称数据集，用于灵巧策略学习。仿真和真实世界实验表明，EgoAERO能够实现单次演示的灵巧操作，并在HOI4D上达到接近基于CAD重建的下游性能。

英文摘要：

Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。