RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Google DeepMind(谷歌DeepMind)
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laur… 展开作者
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, Nicolas Heess
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100-1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.
EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
Shanghai AI Laboratory(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学)
Effective Tuning Strategies for Generalist Robot Manipulation Policies
University of Adelaide(阿德莱德大学);Commonwealth Scientific and Industrial Research Organisation (CSIRO)(联邦科学与工业研究组织(CSIRO));Shanghai Jiao Tong University(上海交通大学);TikTok
GeCCo -- a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots
Dynamic Robot Systems Group, Oxford Robotics Institute, University of Oxford(牛津大学机器人研究所动态机器人系统组);Sony AI(索尼人工智能);Dyson Institute of Engineering and Technology(戴森工程与技术研究所)
GeniWorld: A Generalizable Interactive World Model for Robotic Manipulation via Visual Actions
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tencent Robotics X(腾讯Robotics X实验室);The Hong Kong University of Science and Technology(香港科技大学);Shenzhen Technology University(深圳技术大学)
PrimitiveVLA: Learning Reusable Motion Primitives for Efficient and Generalizable Robotic Manipulation
State Key Lab of Processors, Institute of Computing Technology, CAS(处理器国家重点实验室,计算技术研究所,中国科学院);Jiangsu Key Laboratory of AI for Industries, Institute of AI for Industries, CAS(江苏人工智能工业重点实验室,人工智能工业研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Cambricon Technologies(寒武科技);Intelligent Software Research Center, Institute of Software, CAS(软件研究所智能软件研究中心,中国科学院);University of Science and Technology of China(中国科学技术大学)