Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and improved performance. The proposed algorithm (AWET) weights the critic losses with a novel agent advantage weight to improve over the expert data. In addition, AWET makes use of an automatic early termination technique to stop and discard policy rollouts that are not similar to expert trajectories -- to prevent drifting far from the expert data. In an ablation study, AWET showed improved and promising performance when compared to state-of-the-art baselines on four standard robotic tasks.
We present a method for learning options from segmented demonstration trajectories. The trajectories are first segmented into skills using nonparametric Bayesian clustering and a reward function for each segment is then learned using inverse reinforcement learning. From this, a set of inferred trajectories for the demonstration are generated. Option initiation sets and termination conditions are learned from these trajectories using the one-class support vector machine clustering algorithm. We demonstrate our method in the four rooms domain, where an agent is able to autonomously discover usab…
This paper introduces a novel Learning from Demonstration framework to learn robotic skills with keyframe demonstrations using a Dynamic Bayesian Network (DBN) and a Bayesian Optimized Policy Search approach to improve the learned skills. DBN learns the robot motion, perceptual change in the object of interest (aka skill sub-goals) and the relation between them. The rewards are also learned from the perceptual part of the DBN. The policy search part is a semiblack box algorithm, which we call BO-PI2 . It utilizes the action-perception relation to focus the high-level exploration, uses Gaussian…
Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）
Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by…
One Demonstration Is Enough for Real-World Robotic Reinforcement Learning
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能国家重点实验室）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；PKU-PsiBot Joint Lab（北大-鹏城实验室联合实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations
The University of Hong Kong（香港大学）；JD.com (JingDong)（京东）；Nanyang Technological University (NTU)（南洋理工大学）；Southern University of Science and Technology (SUSTech)（南方科技大学）；Huawei Technologies（华为技术有限公司）；The Chinese University of Hong Kong (CUHK)（香港中文大学）
An option is a short-term skill consisting of a control policy for a specified region of the state space, and a termination condition recognizing leaving that region. In prior work, we proposed an algorithm called Deep Discovery of Options (DDO) to discover options to accelerate reinforcement learning in Atari games. This paper studies an extension to robot imitation learning, called Discovery of Deep Continuous Options (DDCO), where low-level continuous control skills parametrized by deep neural networks are learned from demonstrations. We extend DDO with: (1) a hybrid categorical-continuous …
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；National University of Defense Technology（国防科技大学）；University of Maryland, College Park（马里兰大学帕克分校）
We present a target-driven navigation system to improve mapless visual navigation in indoor scenes. Our method takes a multi-view observation of a robot and a target as inputs at each time step to provide a sequence of actions that move the robot to the target without relying on odometry or GPS at runtime. The system is learned by optimizing a combinational objective encompassing three key designs. First, we propose that an agent conceives the next observation before making an action decision. This is achieved by learning a variational generative module from expert demonstrations. We then prop…
Offline Discovery of Interpretable Skills from Multi-Task Trajectories
Department of Mechanical and Industrial Engineering, and the Operation Research and Reinforcement Learning (DORL) Lab, University of Toronto（机械与工业工程系，以及操作研究与强化学习（DORL）实验室，多伦多大学）；University of Toronto Institute for Aerospace Studies (UTIAS)（多伦多大学航空航天研究所（UTIAS））；Agentic Intelligence Lab, The University of Hong Kong（代理智能实验室，香港大学）
University of California, Irvine（加州大学欧文分校）；University of California, Berkeley（加州大学伯克利分校）
Autonomous agents can learn by imitating teacher demonstrations of the intended behavior. Hierarchical control policies are ubiquitously useful for such learning, having the potential to break down structured tasks into simpler sub-tasks, thereby improving data efficiency and generalization. In this paper, we propose a variational inference method for imitation learning of a control policy represented by parametrized hierarchical procedures (PHP), a program-like structure in which procedures can invoke sub-procedures to perform sub-tasks. Our method discovers the hierarchical structure in a da…
Domain-Adaptive Communication-Rate Optimization for Sim-to-Real Humanoid-Robot Wireless XR Teleoperation
Cooperative Medianet Innovation Center（协作中位网创新中心）；School of Information Science and Electronic Engineering（信息科学与电子工程学院）；Shanghai Jiao Tong University（上海交通大学）
