Contact Trajectory Prompting: In-Context Transfer of Contact-Rich Behaviors from a Single Demonstration
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；TARS Robotics（TARS机器人公司）；National University of Singapore（新加坡国立大学）
Xian Nie, Yujie Zang, Yuhang Zheng, Yupeng Zheng, Songen Gu, Wendi Chen, Chuan Wen, Cewu Lu, Wenchao Ding, Junchi Yan, Shuicheng Yan
In-context robot learning enables robots to acquire new behaviors from demonstrations without test-time parameter updates. However, visual and kinematic specifications alone may not distinguish contact-rich behaviors that share similar motions but require different interaction intensities or patterns. We propose Contact Trajectory Prompting (CTP), a framework for one-shot transfer of contact-rich manipulation behaviors. CTP encodes motion and contact information from a single demonstration into an ordered contact trajectory prompt that captures their temporal evolution and specifies the desired physical interaction behavior. During execution, the policy uses online sensory feedback to retrieve interaction-relevant information from the prompt, conditioning closed-loop action generation on the demonstrated contact behavior. Across real-world wiping, rotation, and insertion tasks on objects excluded from training, CTP achieves 78.7% overall success, outperforming the strongest evaluated baseline by 13.0 percentage points. Changing only the reference demonstration enables CTP to follow contact specifications for both intensity and direction, including an intermediate wiping force level and clockwise or counterclockwise cap rotation. These results demonstrate that contact trajectories serve as effective behavioral specifications, extending in-context robot learning beyond motion imitation to physical interaction transfer. Project website: https://ctp-robotics.github.io/project_page.
Technische Universität Darmstadt（达姆施塔特工业大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Sensor gloves are popular input devices for a large variety of applications including health monitoring, control of music instruments, learning sign language, dexterous computer interfaces, and tele-operating robot hands. Many commercial products as well as low-cost open source projects have been developed. We discuss here how low-cost (approx. 250 EUROs) sensor gloves with force feedback can be build, provide an open source software interface for Matlab and present first results in learning object manipulation skills through imitation learning on the humanoid robot iCub.
VOLT: Vision and Language Trajectory Segmentation for Faster-than-Demonstration Policies
Collab , Dept. of Mechanical Engineering, Virginia Tech, Blacksburg, VA 24061（机械工程系，弗吉尼亚理工学院，布莱克斯堡，VA 24061）；Mitsubishi Electric Research Laboratories ( MERL ), Cambridge, MA 02139（三菱电机研究实验室（MERL），剑桥，MA 02139）
In robotics, there is need of an interactive and expedite learning method as experience is expensive. Robot Learning from Demonstration (RLfD) enables a robot to learn a policy from demonstrations performed by teacher. RLfD enables a human user to add new capabilities to a robot in an intuitive manner, without explicitly reprogramming it. In this work, we present a novel interactive framework, where a collaborative robot learns skills for trajectory based tasks from demonstrations performed by a human teacher. The robot extracts features from each demonstration called as key-points and learns …
MinInter: Minimizing Trajectory Interpolation During Data Augmentation for Imitation Learning
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology（南方科技大学自动化与智能制造学院）；Pengcheng Laboratory（鹏城实验室）；Carnegie Mellon University（卡内基梅隆大学）
Enhanced Robotic Navigation in Deformable Environments using Learning from Demonstration and Dynamic Modulation
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany（慕尼黑机器人与机器智能研究所（MIRMI），慕尼黑技术大学（TUM））；Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学）
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In this paper, we present an intelligent Assistant for Robotic Therapy (iART), that provides robotic assistance during 3D trajectory tracking tasks. We propose a novel LSTM-based robot learning from demonstration (LfD) paradigm to mimic a therapist's assistance behavior. iART presents a trajectory agnostic LfD routine that can generalize learned behavior from a single trajectory to any 3D shape. Once the therapist's behavior has been learned, iART enables the patient to modify this behavior as per their preference. The system requires only a single demonstration of 2 minutes and exhibits a mea…
Learning Forward & Reverse Skills from a Single Unfinished Demonstration for Constrained Manipulation Tasks
Autonomous Systems Lab, Technische Universität Wien (TU Wien)（自主系统实验室，维也纳工业大学（TU Wien））；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（机器人与机电一体化研究所，德国航空航天中心（DLR））
We introduce a simple new method for visual imitation learning, which allows a novel robot manipulation task to be learned from a single human demonstration, without requiring any prior knowledge of the object being interacted with. Our method models imitation learning as a state estimation problem, with the state defined as the end-effector's pose at the point where object interaction begins, as observed from the demonstration. By then modelling a manipulation task as a coarse, approach trajectory followed by a fine, interaction trajectory, this state estimator can be trained in a self-superv…
RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning
Global College, Shanghai Jiao Tong University（上海交通大学Global College）；Shanghai Institute of Physical Intelligence and Robotics (PAIR)（上海物理智能与机器人研究所（PAIR））；Duke Kunshan University（昆山杜克大学）
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the con…
Czech Technical University in Prague（布拉格捷克理工大学）；Inria Paris（法国国家信息与自动化研究所巴黎中心）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）
A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the…
OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
New York University Tandon School of Engineering（纽约大学Tandon工程学院）；New York University Courant Institute of Mathematical Sciences（纽约大学Courant数学科学研究所）；Meta-FAIR
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Thus, they can easily generalize and adapt to new and changing environments. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting, making it difficult for them to imitate human behavior in case of versatile demonstrations. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versat…
Learning from Demonstration depends on a robot learner generalising its learned model to unseen conditions, as it is not feasible for a person to provide a demonstration set that accounts for all possible variations in non-trivial tasks. While there are many learning methods that can handle interpolation of observed data effectively, extrapolation from observed data offers a much greater challenge. To address this problem of generalisation, this paper proposes a modified Task-Parameterised Gaussian Mixture Regression method that considers the relevance of task parameters during trajectory gene…
Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The cano…
Robust Offline Imitation Learning Through State-level Trajectory Stitching
National Key Lab of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China（自主智能无人系统国家重点实验室，北京理工大学，北京100081，中国）；Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China（控制科学与工程学院，哈尔滨工业大学，哈尔滨150001，中国）
University of Surrey（萨里大学）；Karlshule Institute of Technology（卡尔斯鲁厄理工学院）
This work adds on to the on-going efforts to provide more autonomy to space robots. Here the concept of programming by demonstration or imitation learning is used for trajectory planning of manipulators mounted on small spacecraft. For greater autonomy in future space missions and minimal human intervention through ground control, a robot arm having 7-Degrees of Freedom (DoF) is envisaged for carrying out multiple tasks like debris removal, on-orbit servicing and assembly. Since actual hardware implementation of microgravity environment is extremely expensive, the demonstration data for trajec…
College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform novel tasks by providing demonstrations. However, as demonstrators are typically non-experts, modern LfD techniques are unable to produce policies much better than the suboptimal demonstration. A previously-proposed framework, SSRR, has shown success in learning from suboptimal demonstration but relies on noise-injected trajectories to infer an idealized reward function. A random approach such as noise-injection to generate trajectories has two key drawbacks: 1) Perfo…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；New York University（纽约大学）
We present a general, two-stage reinforcement learning approach to create robust policies that can be deployed on real robots without any additional training using a single demonstration generated by trajectory optimization. The demonstration is used in the first stage as a starting point to facilitate initial exploration. In the second stage, the relevant task reward is optimized directly and a policy robust to environment uncertainties is computed. We demonstrate and examine in detail the performance and robustness of our approach on highly dynamic hopping and bounding tasks on a quadruped r…
Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer
Institute of Trustworthy Embodied AI（可信具身人工智能研究所）；Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）；NVIDIA；University of Michigan（密歇根大学）；East China Normal University（华东师范大学）
This paper presents a vision-based learning-by-demonstration approach to enable robots to learn and complete a manipulation task cooperatively. With this method, a vision system is involved in both the task demonstration and reproduction stages. An expert first demonstrates how to use tools to perform a task, while the tool motion is observed using a vision system. The demonstrations are then encoded using a statistical model to generate a reference motion trajectory. Equipped with the same tools and the learned model, the robot is guided by vision to reproduce the task. The task performance w…
Joint Flow Trajectory Optimization For Feasible Robot Motion Generation from Video Demonstrations
College of Connected Computing, Vanderbilt University（连接计算学院，范德比尔特大学）；Robotics Institute, Carnegie Mellon University（机器人研究所，卡内基梅隆大学）；School of Computer Science, The University of Sydney（计算机科学学院，悉尼大学）
