Reinforcement learning (RL) presents a promising framework to learn policies through environment interaction, but often requires an infeasible amount of interaction data to solve complex tasks from sparse rewards. One direction includes augmenting RL with offline data demonstrating desired tasks, but past work often require a lot of high-quality demonstration data that is difficult to obtain, especially for domains such as robotics. Our approach consists of a reverse curriculum followed by a forward curriculum. Unique to our approach compared to past work is the ability to efficiently leverage more than one demonstration via a per-demonstration reverse curriculum generated via state resets. The result of our reverse curriculum is an initial policy that performs well on a narrow initial state distribution and helps overcome difficult exploration problems. A forward curriculum is then used to accelerate the training of the initial policy to perform well on the full initial state distribution of the task and improve demonstration and sample efficiency. We show how the combination of a reverse curriculum and forward curriculum in our method, RFCL, enables significant improvements in demonstration and sample efficiency compared against various state-of-the-art learning-from-demonstration baselines, even solving previously unsolvable tasks that require high precision and control.
Max Planck Institute for Software Systems (MPI-SWS)（马克斯·普朗克软件系统研究所）；Saarland University（萨尔大学）；The Alan Turing Institute（艾伦·图灵研究所）
We consider the problem of teaching via demonstrations in sequential decision-making settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to t…
Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum
Seoul National University（首尔国立大学）；Artificial Intelligence Institute of Seoul National University (AIIS)（首尔国立大学人工智能研究所（AIIS））；Automation and Systems Research Institute (ASRI)（自动化与系统研究所（ASRI））
For reinforcement learning (RL), it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards. To solve this problem, reverse curriculum generation (RCG) provides a reverse expansion approach that automatically generates a curriculum for the agent to learn. More specifically, RCG adapts the initial state distribution from the neighborhood of a goal to a distance as training proceeds. However, the initial state distribution generated for each iteration might be biased, thus making the policy overfit or slowing down the reverse expansion rate. …
Texas A&M University（德克萨斯A&M大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼AI）
Applying reinforcement learning (RL) to sparse reward domains is notoriously challenging due to insufficient guiding signals. Common RL techniques for addressing such domains include (1) learning from demonstrations and (2) curriculum learning. While these two approaches have been studied in detail, they have rarely been considered together. This paper aims to do so by introducing a principled task phasing approach that uses demonstrations to automatically generate a curriculum sequence. Using inverse RL from (suboptimal) demonstrations we define a simple initial task. Our task phasing approac…
Microsoft Research（微软研究院）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
In this paper, we present a technique that improves the process of training an agent (using RL) for instruction following. We develop a training curriculum that uses a nominal number of expert demonstrations and trains the agent in a manner that draws parallels from one of the ways in which humans learn to perform complex tasks, i.e by starting from the goal and working backwards. We test our method on the BabyAI platform and show an improvement in sample efficiency for some of its tasks compared to a PPO (proximal policy optimization) baseline.
University of Freiburg（弗莱堡大学）；Toyota Research Institute（丰田研究院）
We propose Adaptive Curriculum Generation from Demonstrations (ACGD) for reinforcement learning in the presence of sparse rewards. Rather than designing shaped reward functions, ACGD adaptively sets the appropriate task difficulty for the learner by controlling where to sample from the demonstration trajectories and which set of simulation parameters to use. We show that training vision-based control policies in simulation while gradually increasing the difficulty of the task via ACGD improves the policy transfer to the real world. The degree of domain randomization is also gradually increased…
Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration. For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock. These goal-oriented tasks present a considerable challenge for reinforcement learning, since their natural reward function is sparse and prohibitive amounts of exploration are required to reach the goal and receive some learning signal. Past approaches tackle these problems by exploiting expert demonstrations or by manually designing a task-specific reward shaping fu…
Human-Aware Robot Navigation via Reinforcement Learning with Hindsight Experience Replay and Curriculum Learning
The Chinese University of Hong Kong（香港中文大学）；Southern University of Science and Technology（南方科技大学）；Shenzhen Research Institute of the Chinese University of Hong Kong（香港中文大学深圳研究院）
Army Research Laboratory（陆军研究实验室）；Columbia University（哥伦比亚大学）
Traditionally, learning from human demonstrations via direct behavior cloning can lead to high-performance policies given that the algorithm has access to large amounts of high-quality data covering the most likely scenarios to be encountered when the agent is operating. However, in real-world scenarios, expert data is limited and it is desired to train an agent that learns a behavior policy general enough to handle situations that were not demonstrated by the human expert. Another alternative is to learn these policies with no supervision via deep reinforcement learning, however, these algori…
Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
School of Computer Science, Fudan University（复旦大学计算机科学学院）；ByteDance Research（字节跳动研究部）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）
Massachusetts Institute of Technology（麻省理工学院）；Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）
We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipulation tasks with sparse reward functions. Curriculum learning solves complicated RL tasks by introducing a sequence of auxiliary tasks with increasing difficulty, yet how to automatically design effective and generalizable curricula remains a challenging research problem. ACED extracts curricula from a small amount of expert demonstration trajectories by dividing demonstrations …
Multi-agent football poses an unsolved challenge in AI research. Existing work has focused on tackling simplified scenarios of the game, or else leveraging expert demonstrations. In this paper, we develop a multi-agent system to play the full 11 vs. 11 game mode, without demonstrations. This game mode contains aspects that present major challenges to modern reinforcement learning algorithms; multi-agent coordination, long-term planning, and non-transitivity. To address these challenges, we present TiZero; a self-evolving, multi-agent system that learns from scratch. TiZero introduces several i…
Queensland University of Technology（昆士兰科技大学）；CSIRO（联邦科学与工业研究组织）；Monash University（莫纳什大学）
Reliable bipedal walking over complex terrain is a challenging problem, using a curriculum can help learning. Curriculum learning is the idea of starting with an achievable version of a task and increasing the difficulty as a success criteria is met. We propose a 3-stage curriculum to train Deep Reinforcement Learning policies for bipedal walking over various challenging terrains. In the first stage, the agent starts on an easy terrain and the terrain difficulty is gradually increased, while forces derived from a target policy are applied to the robot joints and the base. In the second stage, …
Learning Forward & Reverse Skills from a Single Unfinished Demonstration for Constrained Manipulation Tasks
Autonomous Systems Lab, Technische Universität Wien (TU Wien)（自主系统实验室，维也纳工业大学（TU Wien））；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（机器人与机电一体化研究所，德国航空航天中心（DLR））
Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that …
Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field. Learning to reach such goals is particularly hard without any offline data, expert demonstrations, and reward shaping. In this paper, we propose an algorithm to solve the distant goal-reaching task by using search at training time to automatically generate a curriculum of intermediate states. Our algorithm, Classifier-Planning (C-Planning), frames the learning of the goal-conditioned policie…
SGN-CIRL: Scene Graph-based Navigation with Curriculum, Imitation, and Reinforcement Learning
Moscow Institute of Physics and Technology (MIPT)（莫斯科物理技术学院）；Federal Research Center for Computer Science and Control of Russian Academy of Sciences（俄罗斯科学院计算机科学与控制联邦研究中心）；Artificial Intelligence Research Institute (AIRI)（人工智能研究院）
Offline Imitation Learning with Model-based Reverse Augmentation
National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；Nanjing University（南京大学）；School of Intelligence Science and Technology（智能科学与技术学院）；School of Artificial Intelligence（人工智能学院）
Massachusetts Institute of Technology（麻省理工学院）；University of California Berkeley（加州大学伯克利分校）
Learning robotic manipulation tasks using reinforcement learning with sparse rewards is currently impractical due to the outrageous data requirements. Many practical tasks require manipulation of multiple objects, and the complexity of such tasks increases with the number of objects. Learning from a curriculum of increasingly complex tasks appears to be a natural solution, but unfortunately, does not work for many scenarios. We hypothesize that the inability of the state-of-the-art algorithms to effectively utilize a task curriculum stems from the absence of inductive biases for transferring k…
In robotics, it is often not possible to learn useful policies using pure model-free reinforcement learning without significant reward shaping or curriculum learning. As a consequence, many researchers rely on expert demonstrations to guide learning. However, acquiring expert demonstrations can be expensive. This paper proposes an alternative approach where the solutions of previously solved tasks are used to produce an action prior that can facilitate exploration in future tasks. The action prior is a probability distribution over actions that summarizes the set of policies found solving prev…
Institut des Systèmes Intelligents et de Robotique Sorbonne Université, CNRS UMR 7222（索邦大学智能系统与机器人研究所（法国国家科学研究中心UMR 7222））；FLOWERS Team INRIA Bordeaux Sud Ouest（法国国家信息与自动化研究所波尔多西南 Flowers 团队）
In this paper we study a new reinforcement learning setting where the environment is non-rewarding, contains several possibly related objects of various controllability, and where an apt agent Bob acts independently, with non-observable intentions. We argue that this setting defines a realistic scenario and we present a generic discrete-state discrete-action model of such environments. To learn in this environment, we propose an unsupervised reinforcement learning agent called CLIC for Curriculum Learning and Imitation for Control. CLIC learns to control individual objects in its environment, …
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Compared to reinforcement learning, imitation learning (IL) is a powerful paradigm for training agents to learn control policies efficiently from expert demonstrations. However, in most cases, obtaining demonstration data is costly and laborious, which poses a significant challenge in some scenarios. A promising alternative is to train agent learning skills via imitation learning without expert demonstrations, which, to some extent, would extremely expand imitation learning areas. To achieve such expectation, in this paper, we propose Hindsight Generative Adversarial Imitation Learning (HGAIL)…
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
Eastern Institute of Technology（东技术院）；The Hong Kong Polytechnic University（香港理工大学）；Shanghai Jiao Tong University（上海交通大学）；Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（人工智能 thrust，香港科学与技术大学（广州））
COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphones
Georgia Institute of Technology（佐治亚理工学院）；University of California, Berkeley（加州大学伯克利分校）；New York University Abu Dhabi (NYUAD)（纽约大学阿布扎克分校）；University of Toronto（多伦多大学）；NVIDIA（英伟达）
Theoretical Closed-loop Stability Bounds for Dynamical System Coupled with Diffusion Policies
Department of Mechanical Engineering, Universite de Sherbrooke（机械工程系， Sherbrooke 大学）；Department of Electronic and Computer Engineering, Universite de Sherbrooke（电子与计算机工程系， Sherbrooke 大学）
Robust Path Planning via Learning from Demonstrations for Robotic Catheters in Deformable Environments
Politecnico di Milano（米兰理工大学）；Delft University of Technology（代尔夫特理工大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；Katholieke Universiteit Leuven（鲁汶大学）；Centro Cardiologico Monzino, IRCCS（蒙奇诺心脏病学研究所）
RESTA: Time-Scaling Augmentation for Data-Efficient Robot Imitation Learning
Global College, Shanghai Jiao Tong University（上海交通大学Global College）；Shanghai Institute of Physical Intelligence and Robotics (PAIR)（上海物理智能与机器人研究所（PAIR））；Duke Kunshan University（昆山杜克大学）
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；École de technologie supérieure (ETS)（高等技术学院（ETS））
Reinforcement learning (RL) is a popular machine learning paradigm for game playing, robotics control, and other sequential decision tasks. However, RL agents often have long learning times with high data requirements because they begin by acting randomly. In order to better learn in complex tasks, this article argues that an external teacher can often significantly help the RL agent learn. OpenAI Gym is a common framework for RL research, including a large number of standard environments and agents, making RL research significantly more accessible. This article introduces our new open-source …
Google AI（谷歌人工智能（Google AI））；cs.ucsb.edu（加州大学圣巴巴拉分校计算机科学系（注：此处原机构信息存在混淆，严格按要求提取明确出现的机构，实际“cs.ucsb.edu”是邮箱域名，非独立机构，故仅保留“Google AI”））
Learning in environments with large state and action spaces, and sparse rewards, can hinder a Reinforcement Learning (RL) agent's learning through trial-and-error. For instance, following natural language instructions on the Web (such as booking a flight ticket) leads to RL settings where input vocabulary and number of actionable elements on a page can grow very large. Even though recent approaches improve the success rate on relatively simple environments with the help of human demonstrations to guide the exploration, they still fail in environments where the set of possible instructions can …
Architecture-Agnostic Curriculum Learning for Document Understanding: Empirical Evidence from Text-Only and Multimodal
1 Synchromedia Laboratory, \' E cole de Technologie Sup\' e rieure (\' E TS), 1100 Notre-Dame St W, Montreal, QC H3C 1K3, Canada；2 Department of Computer Science, University of Bari Aldo Moro, Via Orabona 4, 70125 Bari, Italy
