arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23893cs.CVcs.RO
收藏

AoE: 始终在线的以自身为中心的人类视频采集用于具身AI

AoE: Always-on Egocentric Human Video Collection for Embodied AI

Ant Digital Technologies, Ant Group(蚂蚁集团数字技术部);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Zhejiang University(浙江大学);Peking University(北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Bowen Yang, Zishuo Li, Yang Sun, Changtao Miao, Yifan Yang, Man Luo, Xiaotong Yan, Feng Jiang, Jinchuan Shi, Yankai Fu, Ning Chen, Junkai Zhao, Pengwei Wang, Gu… 展开作者

Bowen Yang, Zishuo Li, Yang Sun, Changtao Miao, Yifan Yang, Man Luo, Xiaotong Yan, Feng Jiang, Jinchuan Shi, Yankai Fu, Ning Chen, Junkai Zhao, Pengwei Wang, Guocai Yao, Shanghang Zhang, Hao Chen, Zhe Li, Kai Zhu

AI总结:

AoE系统通过利用人类和智能手机低成本采集以自身为中心的现实世界交互数据,解决具身AI的数据稀缺问题,提升现实世界泛化能力。

摘要:

具身基础模型需要大量、高质量的现实世界交互数据进行预训练和扩展。然而,现有数据收集方法存在基础设施成本高、硬件依赖复杂和交互范围有限等问题,使可扩展性变得困难。事实上,人类本身是理想的物理具身代理。因此,从全球分布的“人类代理”中获取以自身为中心的现实世界交互数据具有低成本和可持续性的优势。为此,我们提出了始终在线的以自身为中心(AoE)数据收集系统,旨在通过利用人类自身及其智能手机来简化硬件依赖,实现低成本、高效率且场景无关的现实世界交互数据收集,以解决数据稀缺的问题。具体而言,我们首先采用一个人体工学的颈部智能手机支架,通过云-边协同架构实现低门槛、大规模的以自身为中心的数据收集。其次,我们开发了一个跨平台移动应用程序,利用设备上的计算能力进行实时处理,同时云端托管自动化标注和过滤流程,将原始视频转化为高质量的训练数据。最后,AoE系统支持任何人、任何时间、任何地点的分布式Ego视频数据收集。我们评估了AoE在数据预处理质量和下游任务上的表现,证明高质量的以自身为中心的数据显著提升了现实世界泛化能力。

英文摘要:

Embodied foundation models require large-scale, high-quality real-world interaction data for pre-training and scaling. However, existing data collection methods suffer from high infrastructure costs, complex hardware dependencies, and limited interaction scope, making scalable expansion challenging. In fact, humans themselves are ideal physically embodied agents. Therefore, obtaining egocentric real-world interaction data from globally distributed "human agents" offers advantages of low cost and sustainability. To this end, we propose the Always-on Egocentric (AoE) data collection system, which aims to simplify hardware dependencies by leveraging humans themselves and their smartphones, enabling low-cost, highly efficient, and scene-agnostic real-world interaction data collection to address the challenge of data scarcity. Specifically, we first employ an ergonomic neck-mounted smartphone holder to enable low-barrier, large-scale egocentric data collection through a cloud-edge collaborative architecture. Second, we develop a cross-platform mobile APP that leverages on-device compute for real-time processing, while the cloud hosts automated labeling and filtering pipelines that transform raw videos into high-quality training data. Finally, the AoE system supports distributed Ego video data collection by anyone, anytime, and anywhere. We evaluate AoE on data preprocessing quality and downstream tasks, demonstrating that high-quality egocentric data significantly boosts real-world generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑