EgoMe：真实世界中基于第一人称视角跟随我的新数据集与挑战
EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World
AI总结：
EgoMe是一个大规模第一人称视角数据集，包含7902对视频，用于研究机器人通过模仿者视角跟随人类行为，并提供基准测试以促进模仿学习研究。
摘要：
在人类模仿学习中，模仿者通常以第一人称视角作为基准，自然地将从第三人称视角观察到的行为迁移到自身，这为研究机器人如何更有效地模仿人类行为提供了灵感。然而，当前研究主要关注不同相机拍摄的自我-外部数据的基本对齐问题，而非从模仿者视角收集数据，这与高级认知过程不一致。为推进该研究，我们引入了一个新颖的大规模第一人称视角数据集，名为EgoMe，旨在通过模仿者在真实世界中的第一人称视角来跟随人类模仿学习的过程。我们的数据集包含7902对配对的外部-自我视频（总计15804个视频），涵盖各种真实场景中的多样日常行为。对于每对视频，一个视频捕捉模仿者观察演示者动作的第三人称视角，另一个视频捕捉模仿者随后跟随这些动作的第一人称视角。值得注意的是，EgoMe独特地整合了外部-自我眼动追踪、其他多模态传感器IMU数据以及不同级别的标注，以帮助建立观察与模仿过程之间的关联。我们进一步提供了一系列具有挑战性的基准测试，以充分利用这一数据资源并促进机器人模仿学习研究。广泛的分析表明，与现有数据集相比，EgoMe具有显著优势。我们的EgoMe数据集和基准测试可在https://huggingface.co/datasets/HeqianQiu/EgoMe获取。
英文摘要：
In human imitation learning, the imitator typically take the egocentric view as a benchmark, naturally transferring behaviors observed from an exocentric view to their owns, which provides inspiration for researching how robots can more effectively imitate human behavior. However, current research primarily focuses on the basic alignment issues of ego-exo data from different cameras, rather than collecting data from the imitator's perspective, which is inconsistent with the high-level cognitive process. To advance this research, we introduce a novel large-scale egocentric dataset, called EgoMe, which towards following the process of human imitation learning via the imitator's egocentric view in the real world. Our dataset includes 7902 paired exo-ego videos (totaling15804 videos) spanning diverse daily behaviors in various real-world scenarios. For each video pair, one video captures an exocentric view of the imitator observing the demonstrator's actions, while the other captures an egocentric view of the imitator subsequently following those actions. Notably, EgoMe uniquely incorporates exo-ego eye gaze, other multi-modal sensor IMU data and different-level annotations for assisting in establishing correlations between observing and imitating process. We further provide a suit of challenging benchmarks for fully leveraging this data resource and promoting the robot imitation learning research. Extensive analysis demonstrates significant advantages over existing datasets. Our EgoMe dataset and benchmarks are available at https://huggingface.co/datasets/HeqianQiu/EgoMe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
