AoE: Always-on Egocentric Human Video Collection for Embodied AI
Ant Digital Technologies, Ant Group(蚂蚁集团数字技术部);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Zhejiang University(浙江大学);Peking University(北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Bowen Yang, Zishuo Li, Yang Sun, Changtao Miao, Yifan Yang, Man Luo, Xiaotong Yan, Feng Jiang, Jinchuan Shi, Yankai Fu, Ning Chen, Junkai Zhao, Pengwei Wang, Gu… 展开作者
Bowen Yang, Zishuo Li, Yang Sun, Changtao Miao, Yifan Yang, Man Luo, Xiaotong Yan, Feng Jiang, Jinchuan Shi, Yankai Fu, Ning Chen, Junkai Zhao, Pengwei Wang, Guocai Yao, Shanghang Zhang, Hao Chen, Zhe Li, Kai Zhu
Embodied foundation models require large-scale, high-quality real-world interaction data for pre-training and scaling. However, existing data collection methods suffer from high infrastructure costs, complex hardware dependencies, and limited interaction scope, making scalable expansion challenging. In fact, humans themselves are ideal physically embodied agents. Therefore, obtaining egocentric real-world interaction data from globally distributed "human agents" offers advantages of low cost and sustainability. To this end, we propose the Always-on Egocentric (AoE) data collection system, which aims to simplify hardware dependencies by leveraging humans themselves and their smartphones, enabling low-cost, highly efficient, and scene-agnostic real-world interaction data collection to address the challenge of data scarcity. Specifically, we first employ an ergonomic neck-mounted smartphone holder to enable low-barrier, large-scale egocentric data collection through a cloud-edge collaborative architecture. Second, we develop a cross-platform mobile APP that leverages on-device compute for real-time processing, while the cloud hosts automated labeling and filtering pipelines that transform raw videos into high-quality training data. Finally, the AoE system supports distributed Ego video data collection by anyone, anytime, and anywhere. We evaluate AoE on data preprocessing quality and downstream tasks, demonstrating that high-quality egocentric data significantly boosts real-world generalization.
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Peking University(北京大学)
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tencent Robotics X(腾讯Robotics X实验室);University of Toronto(多伦多大学);Zhili College, Tsinghua University(清华大学志道书院)
Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
Texas A&M University(德克萨斯A&M大学);University of Minnesota(明尼苏达大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of California, Los Angeles(加州大学洛杉矶分校);University of Texas at Austin(得克萨斯大学奥斯汀分校);Amazon(亚马逊);State University of New York at Stony Brook(纽约州立大学石溪分校)
CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding
JD Explore Academy(京东探索研究院);School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);South China University of Technology(华南理工大学);Sun Yat-sen University(中山大学);School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)人工智能学院)
AtomEgo: Exploring Ego-Robot Integration for Embodied Foundation Model Pretraining
INFIFORCE;Lionrock Artificial Intelligence Laboratory(狮岩人工智能实验室);HKU(香港大学);Tongji University(同济大学);Huazhong University of Science and Technology(华中科技大学)
LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos
Joy Future Academy, JD(京东探索研究院);School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Tsinghua SIGS, Tsinghua University(清华大学深圳国际研究生院);Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学院)
IndustryNav: Evaluating Spatial Reasoning and Safety of Embodied Agents in Dynamic Industrial Navigation
Michigan State University(密歇根州立大学);Independent Researcher(独立研究者);University of California, Irvine(加州大学尔湾分校);Ohio State University(俄亥俄州立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Santa Barbara(加州大学圣巴巴拉分校);University of Pittsburgh(匹兹堡大学);University of Virginia(弗吉尼亚大学);Arizona State University(亚利桑那州立大学);Purdue University Northwest(普渡大学西北分校)
PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Zhongguancun Academy(中关村学院);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院);Harbin Institute of Technology(哈尔滨工业大学);Huazhong University of Science and Technology(华中科技大学)
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Shanghai Jiao Tong University(上海交通大学);Alibaba Group(阿里巴巴集团);Tianji KernalMind Co., Ltd.(天机芯智有限公司);The Hong Kong University of Science and Technology(香港科技大学);Southeast University(东南大学);Renmin University of China(中国人民大学);The University of Tokyo(东京大学)
AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation
University of Arkansas(阿肯色大学);Carnegie Mellon University(卡内基梅隆大学);University of Science, Ho Chi Minh City(胡志明市科学大学);Vietnam National University, Ho Chi Minh City(胡志明市越南国家大学)
VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents
Huawei Heisenberg Research Center(华为海森堡研究中心);Ludwig Maximilian University of Munich(慕尼黑大学);University of Freiburg(弗莱堡大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Department of Applied Physics, Stanford University(斯坦福大学应用物理系);Wu-Tsai Neurosciences Institute, Stanford University(斯坦福大学吴蔡神经科学研究所);Stanford Institute for Human-Centered Artificial Intelligence, Stanford University(斯坦福大学以人为本人工智能研究院)