Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks. However, it is challenging to apply the RL algorithms directly in the real world. For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. For another, it is necessary to make heavy engineering efforts to design reward functions manually. To address these issues, we leverage foundation models in this paper. We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks. Visualizations and code are available at \url{https://yewr.github.io/rlfp}.
The Essential Role of Causality in Foundation World Models for Embodied AI
Microsoft Research(微软研究院);Technical University of Munich(慕尼黑工业大学);KTH royal institute of technology(皇家理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
GenRL: Multimodal-foundation world models for generalization in embodied agents
IDLab, Ghent University(IDLab,根特大学);VERSES AI Research Lab(VERSES AI研究实验室);Mila, University of Montreal(米拉研究所,蒙特利尔大学);ServiceNow Research(ServiceNow研究院)
Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies
University of South Florida(佛罗里达州立大学);Florida International University(佛罗里达国际大学);Johns Hopkins University(约翰霍普金斯大学);Nottingham Trent University(诺丁汉特伦特大学);Imperial College London(伦敦帝国理工学院)
Universal Actions for Enhanced Embodied Foundation Models
AIR, Tsinghua University(清华大学AIR);Sensetime Research(商汤科技研究院);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai AI Lab(上海人工智能实验室);BAAI(北京智源人工智能研究院)
AtomEgo: Exploring Ego-Robot Integration for Embodied Foundation Model Pretraining
INFIFORCE;Lionrock Artificial Intelligence Laboratory(狮岩人工智能实验室);HKU(香港大学);Tongji University(同济大学);Huazhong University of Science and Technology(华中科技大学)
Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL
Peking University(北京大学);National University of Singapore(新加坡国立大学);City University of Macau(澳门城市大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);BIGAI(北京智源人工智能研究院);Beihang University(北京航空航天大学)
Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation
Embodied AI and Robotics (AIR) Lab, New York University, New York, USA and NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE(纽约大学Embodied AI和机器人实验室及纽约大学阿布扎赫尔人工智能与机器人中心);Harvard AI and Robotics Lab, Harvard University, Boston, USA(哈佛大学人工智能与机器人实验室);Department of Computer Science, University College London, London, UK(伦敦大学学院计算机科学系);Department of Computer Science, University of Liverpool, UK(利物浦大学计算机科学系);NYUAD Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE(纽约大学阿布扎赫尔人工智能与机器人中心)
SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hong Kong University of Science and Technology(香港科技大学);Renmin University of China(中国人民大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI))
Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室);Peng Cheng Laboratory(鹏城实验室);Institute of Digital Media, Peking University(北京大学数字媒体研究院)
Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);Beijing Innovation Center of Humanoid Robotics(人形机器人创新中心);The Hong Kong University of Science and Technology(香港科技大学)
EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);AI Lab, Tencent(腾讯AI实验室);Robotics X, Tencent(腾讯Robotics X实验室);Nanjing University of Posts and Telecommunications(南京邮电大学)
Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
Xi’an Jiaotong Univeristy(西安交通大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Chinese Academy of Sciences(中国科学院);Westlake University(西湖大学);Zhejiang University(浙江大学);University of Sydney(悉尼大学);BAAI(百度人工智能研究院);Peking University(北京大学)
Generative Multi-Agent Collaboration in Embodied AI: A Systematic Review
Tongji University(同济大学);East China Normal University(华东师范大学);Technical University of Munich(慕尼黑工业大学);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
School of Software, Harbin Institute of Technology(哈尔滨工业大学软件学院);School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus(赫瑞-瓦特大学马来西亚校区数学与计算机科学学院);School of Future Science and Engineering, Soochow University(苏州大学未来科学与工程学院)
Self-Evolving Cognitive Framework via Causal World Modeling for Embodied Scientific Intelligence
Graduate School of Advanced Science and Engineering, Hiroshima University(广岛大学先进科学与工程研究生院);Advanced Intelligence and Robotics Research Center, Brain Science Institute, Tamagawa University(玉川大学脑科学研究所先进智能与机器人研究中心)
Position: Intelligent Science Laboratory Requires the Integration of Cognitive and Embodied AI
University of Science and Technology of China(中国科学技术大学);Nanjing University(南京大学);University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);Peking University(北京大学);Harbin Institute of Technology(哈尔滨工业大学);Tongji University(同济大学);The University of Sydney, Oxford(悉尼大学,牛津);Renmin University of China(中国人民大学);Swiss Federal Institute of Technology Lausanne(苏黎世联邦理工学院);Shanghai Institute of Ceramics, Chinese Academy of Sciences(中国科学院上海硅酸盐研究所);Shanghai AI Laboratory(上海人工智能实验室);Shanghai AI Laboratory & The Chinese University of Hong Kong(上海人工智能实验室 & 香港中文大学)
Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends
University of Electronic Science and Technology of China(电子科技大学);Qiyuan Lab(启元实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学)
EmbodiSkill: Skill-Aware Reflection for Self-Evolving Embodied Agents
Nanjing University(南京大学);Huazhong University of Science and Technology(华中科技大学);University of Science and Technology of China(中国科学技术大学);Microsoft Research(微软研究院);Institute for AI Industry Research (AIR) Tsinghua University(清华大学人工智能产业研究院)