Traditional model-based reinforcement learning (RL) methods generate forward rollout traces using the learnt dynamics model to reduce interactions with the real environment. The recent model-based RL method considers the way to learn a backward model that specifies the conditional probability of the previous state given the previous action and the current state to additionally generate backward rollout trajectories. However, in this type of model-based method, the samples derived from backward rollouts and those from forward rollouts are simply aggregated together to optimize the policy via the model-free RL algorithm, which may decrease both the sample efficiency and the convergence rate. This is because such an approach ignores the fact that backward rollout traces are often generated starting from some high-value states and are certainly more instructive for the agent to improve the behavior. In this paper, we propose the backward imitation and forward reinforcement learning (BIFRL) framework where the agent treats backward rollout traces as expert demonstrations for the imitation of excellent behaviors, and then collects forward rollout transitions for policy reinforcement. Consequently, BIFRL empowers the agent to both reach to and explore from high-value states in a more efficient manner, and further reduces the real interactions, making it potentially more suitable for real-robot learning. Moreover, a value-regularized generative adversarial network is introduced to augment the valuable states which are infrequently received by the agent. Theoretically, we provide the condition where BIFRL is superior to the baseline methods. Experimentally, we demonstrate that BIFRL acquires the better sample efficiency and produces the competitive asymptotic performance on various MuJoCo locomotion tasks compared against state-of-the-art model-based methods.
National Taiwan University（台湾大学）；University of California San Diego（加利福尼亚大学圣地亚哥分校）；Princeton University（普林斯顿大学）
Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizons, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on th…
On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learning
AI and Robotics Lab, IDLAB-AIRO, Ghent University-imec（人工智能与机器人实验室，IDLAB-AIRO，根特大学-imec）；Robotics, Perception and Learning Lab, (RPL), EECS, KTH Royal Institute of Technology（机器人、感知与学习实验室，（RPL），EECS，皇家理工学院）；INCAR Robotics AB, Stockholm, Sweden（INCAR机器人公司，斯德哥尔摩，瑞典）
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；New Jersey Institute of Technology（新泽西理工学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Translating Flow to Policy via Hindsight Online Imitation
Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；University of California San Diego（加州大学圣地亚哥分校）；Washington University in St. Louis（圣路易斯华盛顿大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Qi Zhi Institute（上海启智研究所）
RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST)（日本国家先进工业科学技术研究院人工智能研究中心）；CNRS-AIST JRL (Joint Robotics Laboratory), IRL（法国国家科学研究中心与日本AIST联合机器人实验室）；Institute for AI and Robotics, Future Robotics Organization, Waseda University（早稻田大学未来机器人组织人工智能与机器人研究所）；AI Robot Association (AIRoA)（人工智能机器人协会）
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the con…
How to Utilize Failure Demo Data?: Effective Data Selection for Imitation Learning Using Distribution Differences in Attention Mechanism
Faculty of Science and Engineering, Waseda University, Tokyo, Japan（科学与工程学部，早稻田大学，东京，日本）；Physical AI Laboratory, Fujitsu Limited, Kanagawa, Japan（物理人工智能实验室， Fujitsu 有限，神奈川，日本）
Khoury College of Computer Sciences（库里计算机学院）；Northeastern University（东北大学）
Behavior cloning of expert demonstrations can speed up learning optimal policies in a more sample-efficient way over reinforcement learning. However, the policy cannot extrapolate well to unseen states outside of the demonstration data, creating covariate shift (agent drifting away from demonstrations) and compounding errors. In this work, we tackle this issue by extending the region of attraction around the demonstrations so that the agent can learn how to get back onto the demonstrated trajectories if it veers off-course. We train a generative backwards dynamics model and generate short imag…
RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning
Global College, Shanghai Jiao Tong University（上海交通大学Global College）；Shanghai Institute of Physical Intelligence and Robotics (PAIR)（上海物理智能与机器人研究所（PAIR））；Duke Kunshan University（昆山杜克大学）
ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
Alibaba Group（阿里巴巴集团）；Beijing Institute of Technology（北京理工大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhongguancun Academy（中关村学院）
Imitation learning is an effective and safe technique to train robot policies in the real world because it does not depend on an expensive random exploration process. However, due to the lack of exploration, learning policies that generalize beyond the demonstrated behaviors is still an open challenge. We present a novel imitation learning framework to enable robots to 1) learn complex real world manipulation tasks efficiently from a small number of human demonstrations, and 2) synthesize new behaviors not contained in the collected demonstrations. Our key insight is that multi-task domains of…
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Sun Yat-sen University（中山大学）；Shenzhen Loop Area Institute（深圳河套学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai AI Laboratory（上海人工智能实验室）；University of Oxford（牛津大学）；University of Science and Technology of China（中国科学技术大学）
Task-Error Residual Learning for Real-Robot Five-Ball Juggling
Technical University of Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森州人工智能中心）
Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and impro…
Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation
Robotics Research Centre, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机器人研究中心，机械与航空航天工程学院，南洋理工大学，新加坡）；Wings Robotics（Wings机器人）
A3RNN: Bi-directional Fusion of Bottom-up and Top-down Process for Developmental Visual Attention in Robots
Department of Intermedia Art and Science, Waseda University（媒体艺术与科学系，早稻田大学）；Research Development Group, Hitachi, Ltd.（日立株式会社研发发展组）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）
Despite its experimental success, Model-based Reinforcement Learning still lacks a complete theoretical understanding. To this end, we analyze the error in the cumulative reward using a contraction approach. We consider both stochastic and deterministic state transitions for continuous (non-discrete) state and action spaces. This approach doesn't require strong assumptions and can recover the typical quadratic error to the horizon. We prove that branched rollouts can reduce this error and are essential for deterministic transitions to have a Bellman contraction. Our analysis of policy mismatch…
WorldSample: Closed-loop Real-robot RL with World Modelling
PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电气与电子工程学院PINE实验室）；Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；School of Automation, Central South University, Changsha, China（中南大学自动化学院）；School of Automation, Beijing University of Posts and Telecommunications, Beijing, China（北京邮电大学自动化学院）
