Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
Shanghai AI Laboratory(上海人工智能实验室);School of Software, Beihang University(北京航空航天大学软件学院);Shanghai Jiao Tong University(上海交通大学);University of Sydney(悉尼大学)
Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation
NAVER LABS Europe(NAVER欧洲实验室);TU Darmstadt(达姆施塔特工业大学);EPITA Research Laboratory (LRE)(EPITA研究实验室);German Research Center for AI (DFKI)(德国人工智能研究中心);Center for Cognitive Science, TU Darmstadt(达姆施塔特工业大学认知科学中心);Hessian Center for Artificial Intelligence (Hessian.ai)(黑森州人工智能中心)
Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China(人工智能学院,智能与计算学院,天津大学,中国);School of Computer Science and Information Engineering, Hefei University of Technology, China(计算机科学与信息工程学院,合肥工业大学,中国)
本文提出Decompose and Recompose框架,通过分解现有任务演示为可解释的技能-动作对,实现跨任务机械操作的零样本泛化。
ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware Task-specific Intuitive Physics
Department of Computer Science, Rice University, Houston, TX 77005, USA(计算机科学系,里士大学,德克萨斯州休斯敦,77005,美国);Robotics and AI Institute, Cambridge, MA 02142, USA(机器人与人工智能研究所,马萨诸塞州剑桥,02142,美国)
RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics
Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院,清华大学);YXGN Robotics(YXGN机器人);China University of Mining and Technology(中国矿业大学);Shenzhen Technology University(深圳技术大学);Huazhong University of Science and Technology(华中科技大学);Xiamen University(厦门大学)
E-TTS: A New Embodied Test-Time Scaling Framework for Robotic Manipulation
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);FiveAges(五时代)
Generalizable Skill Learning for Construction Robots with Crowdsourced Natural Language Instructions, Composable Skills Standardization, and Large Language Model
Virginia Tech(弗吉尼亚理工大学);University of Michigan(密歇根大学)
OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
National University of Singapore(新加坡国立大学);Tsinghua University(清华大学);Peking University(北京大学);The University of Hong Kong(香港大学);Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)