发表机构
The University of Hong Kong; Kinetix AI（香港大学; Kinetix AI）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
EgoWild2Dex利用真实世界以自我为中心的人类数据，通过GeoFormer对齐视点和动作，实现双臂灵巧操作，在真实机器人上达到96.7%成功率，并发布538.9小时数据集。
AI 中文摘要
以自我为中心的人类数据为学习灵巧机器人操作提供了原则性的监督来源。与先前通常在受限或专门构建的环境中收集此类数据的方法不同，我们在真实世界环境中收集真实世界中的以自我为中心的演示，包括家庭、工厂和药房等，人们在佩戴头戴式相机的同时执行日常任务。这种收集协议捕获了跨长尾对象和技能分布的多样化工作流程和手-物交互，但也由于场景杂乱和头部运动引起的视点变化（平均累积旋转为每秒15.93度）而产生视觉上具有挑战性的观察。为了解决这些问题，我们引入了EgoWild2Dex，它通过分别将不稳定的以自我为中心的视图和人体运动与机器人观察和动作对齐，将真实世界中的以自我为中心的人类经验迁移到具有灵巧双手的双臂机器人上。这项工作提供了三个优势。首先，我们引入了GeoFormer，一种可微分的几何变换器，将嘈杂的人类观察扭曲为机器人观察。其次，我们设计了一种人-机器人训练方案来弥合具身差距，从而在有限的机器人监督下实现高任务成功率。第三，我们发布了EgoWild，一个538.9小时的真实世界以自我为中心的人类数据集，包含179,049个片段、125,961个独特任务描述和1,282个对象类别。在真实机器人上，EgoWild2Dex在三个长视野双手灵巧操作任务中实现了96.7%的平均成功率，以及33.3%的平均对象级零样本成功率。数据、模型和代码将发布。
英文摘要
Egocentric human data provide a principled source of supervision for learning dexterous robot manipulation. Unlike prior approaches that often collect such data in constrained or specially constructed environments, we collect in-the-wild egocentric demonstrations in real-world settings, including homes, factories, and pharmacies, etc., where people perform their ordinary tasks while wearing head-mounted cameras. This collection protocol captures diverse workflows and hand-object interactions across long-tailed object and skill distributions, but also yields visually challenging observations due to scene clutter and head-motion-induced viewpoint changes (a mean cumulative rotation of $15.93^{\circ}$/s). To address these issues, we introduce EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, respectively. This work offers three benefits. First, we introduce GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot observations. Second, we design a human-robot training scheme to bridge the embodiment gap, enabling high task success with limited robot supervision. Third, we release EgoWild, a 538.9-hour in-the-wild egocentric human dataset comprising 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. On real robots, EgoWild2Dex achieves an average success rate of 96.7% across three long-horizon bimanual dexterous manipulation tasks and an average object-level zero-shot success rate of 33.3%. The data, models, and code will be released.
