We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.
Self-Evolving Cognitive Framework via Causal World Modeling for Embodied Scientific Intelligence
Graduate School of Advanced Science and Engineering, Hiroshima University(广岛大学先进科学与工程研究生院);Advanced Intelligence and Robotics Research Center, Brain Science Institute, Tamagawa University(玉川大学脑科学研究所先进智能与机器人研究中心)
Embodied Intelligence for Flexible Manufacturing: A Survey
National University of Defense Technology(中国人民解放军国防科技大学);Wuhan University(武汉大学);Shenzhen University(深圳大学);Hunan University(湖南大学);Shenyang Institute of Automation Chinese Academy of Sciences(中国科学院沈阳自动化研究所)
UAVs Meet Embodied Intelligence: Bridging Human Intents and Flying Dynamics Via Harnessing Physical-Digital AI Agents
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beihang University(北京航空航天大学);Shanxi Datong University(山西大同大学);Johns Hopkins University(约翰斯·霍普金斯大学);University of Sanya(三亚学院);Wenzhou-Kean University(温州肯恩大学);Macau University of Science and Technology(澳门科技大学);KOKONI 3D, Moxin Technology(魔芯科技KOKONI 3D);University of Science and Technology of China(中国科学技术大学)
College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics(南京航空航天大学电子与信息工程学院);College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院)
From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
Massachusetts Institute of Technology(麻省理工学院);University of Oxford(牛津大学);University of Toronto(多伦多大学);Waverly;Mila;University of Montreal(蒙特利尔大学);Stanford University(斯坦福大学);Technische Universität Berlin(柏林工业大学);Element AI – A ServiceNow Company(Element AI – ServiceNow旗下公司);University of Washington(华盛顿大学);University of Pennsylvania(宾夕法尼亚大学);McGill University(麦吉尔大学);[Google] X
Towards Shared Embodied Intelligence in Humanoid Robots through Optimization Development and Testing of the Human Aware ergoCub Robot
GenerativeBionics(生成生物技术);Artificial and Mechanical Intelligence(人工与机械智能);School of Computer Science, University of Manchester(曼彻斯特大学计算机科学学院);Humanoid Sensing and Perception(人形感知与感知);iCub Tech Facility(iCub技术设施);DiMEILA, Istituto Nazionale Assicurazione Infortuni sul Lavoro (INAIL)(DiMEILA,意大利国家职业伤害保险机构(INAIL))
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
Nanjing University(南京大学);University of Hong Kong(香港大学);Central South University(中南大学);Horizon Robotics(Horizon机器人);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Jiao Tong University(上海交通大学);Technical University of Munich(慕尼黑技术大学);Tsinghua University(清华大学)
Embodied Artificial Intelligence through Distributed Adaptive Control: An Integrated Framework
Universitat Pompeu Fabra(庞培法布拉大学);ICREA(加泰罗尼亚高等研究院);Institute for Bioengineering of Catalonia (IBEC)(加泰罗尼亚生物工程研究所);Barcelona Institute of Science and Technology (BIST)(巴塞罗那科学技术研究所)
Enhancing Surgical Robots with Embodied Intelligence for Autonomous Ultrasound Scanning
Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Department of Applied Physics, Stanford University(斯坦福大学应用物理系);Wu-Tsai Neurosciences Institute, Stanford University(斯坦福大学吴蔡神经科学研究所);Stanford Institute for Human-Centered Artificial Intelligence, Stanford University(斯坦福大学以人为本人工智能研究院)
Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents
College of Mechanical Engineering, Chongqing University of Technology(重庆理工大学机械工程学院);James Watt School of Engineering, University of Glasgow(格拉斯哥大学詹姆斯·瓦特工程学院);School of Energy and Power, Jiangsu University of Science and Technology(江苏科技大学能源与动力学院);Magnesium Research Center, Kumamoto University(熊本大学镁研究中心);Department of Information and Communication Engineering, Nagoya University(名古屋大学信息与通信工程系);State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University(浙江大学流体动力与机电系统国家重点实验室)
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室);Peng Cheng Laboratory(鹏城实验室);Institute of Digital Media, Peking University(北京大学数字媒体研究院)
Agentic AI-Empowered Conversational Embodied Intelligence Networks in 6G
Key Laboratory of Broadband Wireless Communication and Sensor Network Technology(宽带无线通信与传感网络技术重点实验室);Nanjing University of Posts and Telecommunications(南京邮电大学);College of Technological Innovation(技术创新学院)
From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
Department of Computer Science and Technology, Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, THBI Lab(计算机科学与技术系、人工智能研究院、BNRist中心、清华-博世联合机器学习中心、THBI实验室);Institute of Artificial Intelligence(人工智能研究院);Department of Psychological and Cognitive Sciences(心理学与认知科学系)
Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimization
X-Humanoid;Imperial College London(帝国理工学院);Peking University(北京大学);University of Manchester(曼彻斯特大学);Westlake University(西湖大学);Fudan University(复旦大学);Waseda University(早稻田大学);Nvidia;Queen Mary University of London(伦敦大学玛丽女王学院);Celonis AI;Meta AI