Large Language Models (LLMs) have recently shown promise as high-level planners for robots when given access to a selection of low-level skills. However, it is often assumed that LLMs do not possess sufficient knowledge to be used for the low-level trajectories themselves. In this work, we address this assumption thoroughly, and investigate if an LLM (GPT-4) can directly predict a dense sequence of end-effector poses for manipulation tasks, when given access to only object detection and segmentation vision models. We designed a single, task-agnostic prompt, without any in-context examples, motion primitives, or external trajectory optimisers. Then we studied how well it can perform across 30 real-world language-based tasks, such as "open the bottle cap" and "wipe the plate with the sponge", and we investigated which design choices in this prompt are the most important. Our conclusions raise the assumed limit of LLMs for robotics, and we reveal for the first time that LLMs do indeed possess an understanding of low-level robot control sufficient for a range of common tasks, and that they can additionally detect failures and then re-plan trajectories accordingly. Videos, prompts, and code are available at: https://www.robot-learning.uk/language-models-trajectory-generators.
Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zerith Robotics(智臻机器人);University of Science and Technology of China(中国科学技术大学)
OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
New York University Tandon School of Engineering(纽约大学Tandon工程学院);New York University Courant Institute of Mathematical Sciences(纽约大学Courant数学科学研究所);Meta-FAIR
Multiple chaotic central pattern generators with learning for legged locomotion and malfunction compensation
Beijing University of Aeronautics and Astronautics(北京航空航天大学);Bernstein Center for Computational Neuroscience(伯恩斯坦计算神经科学中心);Georg-August-Universität Göttingen(哥廷根大学);University of Southern Denmark(南丹麦大学)
Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotion
Chinese University of Hong Kong(香港中文大学);Baidu Inc.(百度公司);Southern University of Science and Technology(南方科技大学);Shenzhen Research Institute of the Chinese University of Hong Kong(香港中文大学深圳研究院)
LongComp: Long-Tail Compositional Zero-Shot Generalization for Robust Trajectory Prediction
School of Computer Science, Carnegie Mellon University(计算机科学系,卡内基梅隆大学);Stack AV Research Internship(Stack AV研究实习);Bosch Center for Artificial Intelligence(博世人工智能中心)
Stereo-Planner: Learning Stereo-to-Trajectory Planning for Zero-Shot Sim-to-Real Quadrotor Navigation
School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院);School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院)
Prompting Decision Transformer for Few-Shot Policy Generalization
Carnegie Mellon University(卡内基梅隆大学);University of Montreal(蒙特利尔大学);Mila(米拉);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Massachusetts Institute of Technology(麻省理工学院);UMass Amherst(马萨诸塞大学阿默斯特分校)
SocialNav-Map: Dynamic Mapping with Human Trajectory Prediction for Zero-Shot Social Navigation
Peng Cheng Laboratory(鹏城实验室);Xiaomi EV(小米汽车);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));National University of Singapore(新加坡国立大学)
DAG-STL: A Hierarchical Framework for Zero-Shot Trajectory Planning under Signal Temporal Logic Specifications
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China(自动化与智能感知学院,上海交通大学,上海,中国);Institute of Artificial Intelligence, Xiamen University, Xiamen, China(人工智能研究院,厦门大学,厦门,中国)
Learning On The Job: Zero-Shot Task Execution under Parametric Uncertainty via Trajectory-Parametrized Dual Control
Department of Electromechanical, Systems and Metal Engineering, Ghent University(机电系统与金属工程系,根特大学);Department of Mechanical Engineering, Yale University(机械工程系,耶鲁大学)