EgoDex: 从大规模第一人称视频中学习灵巧操作
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
AI总结：
EgoDex通过大规模第一人称视频数据集推动灵巧操作研究，提供3D手部跟踪数据及模仿学习评估框架。
摘要：
模仿学习在操作领域存在众所周知的数据稀缺问题。与自然语言和2D计算机视觉不同，目前没有互联网级别的数据集用于灵巧操作。一个有吸引力的选项是第一人称人类视频，这是一种被动可扩展的数据源。然而，现有大规模数据集如Ego4D没有原生的手部姿态标注，也不专注于物体操作。为此，我们使用Apple Vision Pro收集EgoDex：到目前为止最大的、最多样化的灵巧人类操作数据集。EgoDex包含829小时的第一人称视频，配对的3D手和手指跟踪数据是在录制时收集的，其中可以使用多个校准相机和设备SLAM来精确跟踪每只手每个关节的姿态。该数据集涵盖了日常家居物品在194种不同桌面任务中的广泛多样操作行为，从系鞋带到叠衣服。此外，我们训练并系统评估了在数据集上进行手部轨迹预测的模仿学习策略，引入了衡量该日益重要领域进展的指标和基准。通过发布这个大规模数据集，我们希望推动机器人学、计算机视觉和基础模型的前沿。EgoDex可在https://github.com/apple/ml-egodex上公开下载。
英文摘要：
Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download at https://github.com/apple/ml-egodex.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
