We explore how to enhance next-token prediction models to perform in-context imitation learning on a real robot, where the robot executes new tasks by interpreting contextual information provided during the input phase, without updating its underlying policy parameters. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories without relying on any linguistic data or reward function. This formulation enables flexible and training-free execution of new tasks at test time, achieved by prompting the model with sensorimotor trajectories of the new task composing of image observations, actions and states tuples, collected through human teleoperation. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data. In a multitask environment setup, ICRT significantly outperforms current state-of-the-art next-token prediction models in robotics on generalizing to unseen tasks. Code, checkpoints and data are available on https://icrt.dev/
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
University of Technology Sydney(悉尼科技大学);Australia Artificial Intelligence Institute(澳大利亚人工智能研究院);School of Computer Science(计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
T2S: Tokenized Skill Scaling for Lifelong Imitation Learning
School of Computer Science and Technology, East China Normal University, Shanghai 200062, China(东华大学计算机科学与技术学院);Shanghai Innovation Institute, Shanghai 200062, China(上海创新研究院);School of Informatics, Xiamen University, Xiamen, Fujian 361005, China(厦门大学信息学院)
WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
School of Remote Sensing and Information Engineering, Wuhan University(武汉大学遥感信息工程学院);Tsinghua University(清华大学);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院)
Phase-Conditioned Imitation Learning with Autonomous Failure Recovery for Robust Deformable Object Manipulation
Department of Robotics, Tohoku University(东大理学院机器人系);JC STEM Lab of Robotics for Soft Materials, the Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Hong Kong(香港大学工程学院电气与计算机工程系软材料机器人实验室)
Fighting Copycat Agents in Behavioral Cloning from Observation Histories
UC Berkeley(加州大学伯克利分校);University of Pennsylvania(宾夕法尼亚大学);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qi Zhi Institute(上海期智研究院)