Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Jan Achterhold, Markus Krimmel, Joerg Stueckler
备注：
Project website (including video) is available at https://seads.is.tue.mpg.de/. (v2) Accepted for publication at the 6th Conference on Robot Learning (CoRL) 2022, Auckland, New Zealand. (v3) Added details on checkpointing (S.8.1), with references on p.7, p.8, p.21 to clarify number of env. steps of reported results
需要同时具备长时程规划和连续控制能力的问题，对现有强化学习智能体构成了重大挑战。本文提出一种新颖的分层强化学习智能体，将用于连续控制的时间扩展技能与环境状态符号离散抽象中的前向模型相连接，以进行规划。我们将该智能体命名为 SEADS，即 Symbolic Effect-Aware Diverse Skills（符号效应感知的多样化技能）。我们构建了一个目标函数及相应算法，在给定已知状态抽象的情况下，通过内在动机无监督地学习一组多样化技能。这些技能与符号前向模型联合学习，该模型捕捉在状态抽象中执行技能所产生的效应。训练完成后，我们可以利用前向模型将这些技能作为符号动作来进行长时程规划，随后使用所学的连续动作控制技能执行该计划。所提出的算法能够学习技能和前向模型，可用于以高成功率解决同时需要连续控制和长时程规划能力的复杂任务。与其他扁平式和分层强化学习基线智能体相比，该方法表现更优，并在真实机器人上得到成功演示。
英文摘要：
Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment's state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.
The ability to effectively reuse prior knowledge is a key requirement when building general and flexible Reinforcement Learning (RL) agents. Skill reuse is one of the most common approaches, but current methods have considerable limitations.For example, fine-tuning an existing policy frequently fails, as the policy can degrade rapidly early in training. In a similar vein, distillation of expert behavior can lead to poor results when given sub-optimal experts. We compare several common approaches for skill transfer on multiple domains including changes in task and system dynamics. We identify h…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；IIIS, Tsinghua（清华大学智能产业研究院）；NVIDIA Research（英伟达研究院）
Imitation learning offers a promising path for robots to learn general-purpose behaviors, but traditionally has exhibited limited scalability due to high data supervision requirements and brittle generalization. Inspired by recent advances in multi-task imitation learning, we investigate the use of prior data from previous tasks to facilitate learning novel tasks in a robust, data-efficient manner. To make effective use of the prior data, the robot must internalize knowledge from past experiences and contextualize this knowledge in novel tasks. To that end, we develop a skill-based imitation l…
Lund University（隆德大学）；University of Patras（帕特雷大学）；Stanford University（斯坦福大学）
In modern industrial settings with small batch sizes it should be easy to set up a robot system for a new task. Strategies exist, e.g. the use of skills, but when it comes to handling forces and torques, these systems often fall short. We introduce an approach that provides a combination of task-level planning with targeted learning of scenario-specific parameters for skill-based systems. We propose the following pipeline: (1) the user provides a task goal in the planning language PDDL, (2) a plan (i.e., a sequence of skills) is generated and the learnable parameters of the skills are automati…
Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning an…
Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learning
Intelligent Autonomous Systems, TU Darmstadt, Germany（图腾达姆施塔特大学智能自主系统研究所）；German Research Center for AI, Germany（德国人工智能研究中心）；Hessian Centre for Artificial Intelligence, Germany（黑森州人工智能中心）；Honda Research Institute EU, Germany（本田欧洲研究院）
Layer 6 AI（Layer 6人工智能公司）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Skills or low-level policies in reinforcement learning are temporally extended actions that can speed up learning and enable complex behaviours. Recent work in offline reinforcement learning and imitation learning has proposed several techniques for skill discovery from a set of expert trajectories. While these methods are promising, the number K of skills to discover is always a fixed hyperparameter, which requires either prior knowledge about the environment or an additional parameter search to tune it. We first propose a method for offline learning of options (a particular skill framework) …
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
Decision-making is challenging in robotics environments with continuous object-centric states, continuous actions, long horizons, and sparse feedback. Hierarchical approaches, such as task and motion planning (TAMP), address these challenges by decomposing decision-making into two or more levels of abstraction. In a setting where demonstrations and symbolic predicates are given, prior work has shown how to learn symbolic operators and neural samplers for TAMP with manually designed parameterized policies. Our main contribution is a method for learning parameterized polices in combination with …
In this paper, we study the problem of learning a repertoire of low-level skills from raw images that can be sequenced to complete long-horizon visuomotor tasks. Reinforcement learning (RL) is a promising approach for acquiring short-horizon skills autonomously. However, the focus of RL algorithms has largely been on the success of those individual skills, more so than learning and grounding a large repertoire of skills that can be sequenced to complete extended multi-stage tasks. The latter demands robustness and persistence, as errors in skills can compound over time, and may require the rob…
Czech Technical University in Prague（布拉格捷克理工大学）；IIIT Hyderabad（海得拉巴国际信息技术学院）
We aim to teach robots to perform simple object manipulation tasks by watching a single video demonstration. Towards this goal, we propose an optimization approach that outputs a coarse and temporally evolving 3D scene to mimic the action demonstrated in the input video. Similar to previous work, a differentiable renderer ensures perceptual fidelity between the 3D scene and the 2D video. Our key novelty lies in the inclusion of a differentiable approach to solve a set of Ordinary Differential Equations (ODEs) that allows us to approximately model laws of physics such as gravity, friction, and …
University of Freiburg（弗赖堡大学）；University of Technology Nuremberg（纽伦堡工业大学）
Everyday tasks of long-horizon and comprising a sequence of multiple implicit subtasks still impose a major challenge in offline robot control. While a number of prior methods aimed to address this setting with variants of imitation and offline reinforcement learning, the learned behavior is typically narrow and often struggles to reach configurable long-horizon goals. As both paradigms have complementary strengths and weaknesses, we propose a novel hierarchical approach that combines the strengths of both methods to learn task-agnostic long-horizon policies from high-dimensional camera observ…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）
Many real-world reinforcement learning (RL) problems necessitate learning complex, temporally extended behavior that may only receive reward signal when the behavior is completed. If the reward-worthy behavior is known, it can be specified in terms of a non-Markovian reward function - a function that depends on aspects of the state-action history, rather than just the current state and action. Such reward functions yield sparse rewards, necessitating an inordinate number of experiences to find a policy that captures the reward-worthy pattern of behavior. Recent work has leveraged Knowledge Rep…
Neuro-Symbolic Skill Discovery for Conditional Multi-Level Planning
Department of Computer Science and Technology The University of Cambridge（计算机科学与技术系 剑桥大学）；Department of Computer Engineering Bogazici University（计算机工程系 boazici大学）；SISREC Osaka University（Osaka大学SISREC）
Mila（米拉人工智能研究院）；Université de Montréal（蒙特利尔大学）；Amii（阿尔伯塔机器智能研究所）；University of Alberta（阿尔伯塔大学）；McGill University（麦吉尔大学）；Meta AI
In reinforcement learning, the graph Laplacian has proved to be a valuable tool in the task-agnostic setting, with applications ranging from skill discovery to reward shaping. Recently, learning the Laplacian representation has been framed as the optimization of a temporally-contrastive objective to overcome its computational limitations in large (or continuous) state spaces. However, this approach requires uniform access to all states in the state space, overlooking the exploration problem that emerges during the representation learning process. In this work, we propose an alternative method …
Automated discovery of symbolic laws governing skill acquisition from naturally occurring data
National Engineering Research Center of Educational Big Data, Central China Normal University（华中师范大学教育大数据国家工程研究中心）；Faculty of Artificial Intelligence in Education, Central China Normal University（华中师范大学人工智能教育学部）；National Engineering Research Center for E-learning, Central China Normal University（华中师范大学国家数字化学习工程技术研究中心）
The options framework in reinforcement learning models the notion of a skill or a temporally extended sequence of actions. The discovery of a reusable set of skills has typically entailed building options, that navigate to bottleneck states. This work adopts a complementary approach, where we attempt to discover options that navigate to landmark states. These states are prototypical representatives of well-connected regions and can hence access the associated region with relative ease. In this work, we propose Successor Options, which leverages Successor Representations to build a model of the…
La Sapienza University of Rome（拉维亚大学罗马分校）；Peoples' Friendship University of Russia (RUDN University)（俄罗斯人民友谊大学）；Joint Institute for Nuclear Research（联合核子研究所）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；University of Skövde（斯德哥尔摩大学）
The Technion - Israel Institute of Technology（以色列理工学院）；Google Deepmind（谷歌DeepMind）
We introduce the Adaptive Skills, Adaptive Partitions (ASAP) framework that (1) learns skills (i.e., temporally extended actions or options) as well as (2) where to apply them. We believe that both (1) and (2) are necessary for a truly general skill learning framework, which is a key building block needed to scale up to lifelong learning agents. The ASAP framework can also solve related new tasks simply by adapting where it applies its existing learned skills. We prove that ASAP converges to a local optimum under natural conditions. Finally, our experimental results, which include a RoboCup do…
Reinforcement learning has the potential to automate the acquisition of behavior in complex settings, but in order for it to be successfully deployed, a number of practical challenges must be addressed. First, in real world settings, when an agent attempts a task and fails, the environment must somehow "reset" so that the agent can attempt the task again. While easy in simulation, this could require considerable human effort in the real world, especially if the number of trials is very large. Second, real world learning often involves complex, temporally extended behavior that is often difficu…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Procedural planning aims to implement complex high-level goals by decomposition into sequential simpler low-level steps. Although procedural planning is a basic skill set for humans in daily life, it remains a challenge for large language models (LLMs) that lack a deep understanding of the cause-effect relations in procedures. Previous methods require manual exemplars to acquire procedural planning knowledge from LLMs in the zero-shot setting. However, such elicited pre-trained knowledge in LLMs induces spurious correlations between goals and steps, which impair the model generalization to uns…
GenMem: Generative Symbolic Memory for Self-Evolving Harness
National Engineering Research Center of Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（教育部高可信软件技术重点实验室）；Center on Frontiers of Computing Studies, Peking University（北京大学计算前沿研究中心）
