从过往经验中发现行为并将其迁移到新任务的能力，是智能体在现实世界中以高样本效率行动的标志。赋予具身强化学习器同样的能力，可能对其在机器人领域的成功部署至关重要。尽管层级强化学习和KL正则化强化学习各自在此方面展现出前景，但可以说，一种混合方法能够结合二者各自的优势。这些领域的关键在于利用跨架构模块的信息不对称，以偏向所学技能的内容。虽然不对称性的选择对可迁移性有很大影响，但现有方法主要基于直觉，以与领域无关且可能次优的方式作出选择。本文从理论和实证上展示了跨序列任务中技能在表达能力与可迁移性之间的关键权衡，该权衡由信息不对称控制。基于这一洞见，我们提出了Attentive Priors for Expressive and Transferable Skills（APES，面向表达性与可迁移技能的注意力先验），这是一种层级化KL正则化方法，充分受益于先验和层级结构。与现有方法不同，APES基于我们的表达能力—可迁移性定理，以数据驱动、依赖领域的方式学习不对称性，从而使该选择自动化。在具有不同外推程度和稀疏性的复杂迁移领域（例如机器人积木堆叠）上的实验，证明了正确的不对称选择的关键性，并且APES大幅优于以往方法。
英文摘要：
The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt. While asymmetry choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially sub-optimal, manner. In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry. Given this insight, we introduce Attentive Priors for Expressive and Transferable Skills (APES), a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy. Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain-dependent, way based on our expressivity-transferability theorems. Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.
We describe a framework for building abstraction hierarchies whereby an agent alternates skill- and representation-acquisition phases to construct a sequence of increasingly abstract Markov decision processes. Our formulation builds on recent results showing that the appropriate abstract representation of a problem is specified by the agent's skills. We describe how such a hierarchy can be used for fast planning, and illustrate the construction of an appropriate hierarchy for the Taxi domain.
Deep reinforcement learning (DRL) is capable of learning high-performing policies on a variety of complex high-dimensional tasks, ranging from video games to robotic manipulation. However, standard DRL methods often suffer from poor sample efficiency, partially because they aim to be entirely problem-agnostic. In this work, we introduce a novel approach to exploration and hierarchical skill learning that derives its sample efficiency from intuitive assumptions it makes about the behavior of objects both in the physical world and simulations which mimic physics. Specifically, we propose the Hyp…
Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new …
School of Engineering & IT University of New South Wales Canberra（新南威尔士大学堪培拉校区工程与信息技术学院）
The design of reward functions in reinforcement learning is a human skill that comes with experience. Unfortunately, there is not any methodology in the literature that could guide a human to design the reward function or to allow a human to transfer the skills developed in designing reward functions to another human and in a systematic manner. In this paper, we use Systematic Instructional Design, an approach in human education, to engineer a machine education methodology to design reward functions for reinforcement learning. We demonstrate the methodology in designing a hierarchical genetic …
We propose a novel framework and algorithm for hierarchical planning based on the principle of delegation. This framework, the Markov Intent Process, features a collection of skills which are each specialised to perform a single task well. Skills are aware of their intended effects and are able to analyse planning goals to delegate planning to the best-suited skill. This principle dynamically creates a hierarchy of plans, in which each skill plans for sub-goals for which it is specialised. The proposed planning method features on-demand execution---skill policies are only evaluated when needed…
Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Microsoft Research Montréal（微软研究院蒙特利尔）；Université de Montréal（蒙特利尔大学）
A modular design encourages neural models to disentangle and recombine different facets of knowledge to generalise more systematically to new tasks. In this work, we assume that each task is associated with a subset of latent discrete skills from a (potentially small) inventory. In turn, skills correspond to parameter-efficient (sparse / low-rank) model parameterisations. By jointly learning these and a task-skill allocation matrix, the network for each task is instantiated as the average of the parameters of active skills. To favour non-trivial soft partitions of skills across tasks, we exper…
Autonomous agents situated in real-world environments must be able to master large repertoires of skills. While a single short skill can be learned quickly, it would be impractical to learn every task independently. Instead, the agent should share knowledge across behaviors such that each task can be learned efficiently, and such that the resulting model can generalize to new tasks, especially ones that are compositions or subsets of tasks seen previously. A policy conditioned on a goal or demonstration has the potential to share knowledge between tasks if it sees enough diversity of inputs. H…
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Artificial Intelligence Research Institute（人工智能研究所）
This work studies object goal navigation task, which involves navigating to the closest object related to the given semantic category in unseen environments. Recent works have shown significant achievements both in the end-to-end Reinforcement Learning approach and modular systems, but need a big step forward to be robust and optimal. We propose a hierarchical method that incorporates standard task formulation and additional area knowledge as landmarks, with a way to extract these landmarks. In a hierarchy, a low level consists of separately trained algorithms to the most intuitive skills, and…
The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be resolved, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty and skill…
Deep reinforcement learning yields great results for a large array of problems, but models are generally retrained anew for each new problem to be solved. Prior learning and knowledge are difficult to incorporate when training new models, requiring increasingly longer training as problems become more complex. This is especially problematic for problems with sparse rewards. We provide a solution to these problems by introducing Concept Network Reinforcement Learning (CNRL), a framework which allows us to decompose problems using a multi-level hierarchy. Concepts in a concept network are reusabl…
In recent years, adaptive learning systems rely increasingly on learning hierarchy to customize the educational logic developed in their courses. Most approaches do not consider that the relationships of prerequisites between the skills are fuzzy relationships. In this article, we describe a new approach of a practical application of fuzzy logic techniques to the construction of learning hierarchies. For this, we use a learning hierarchy predefined by one or more experts of a specific field. However, the relationships of prerequisites between the skills in the learning hierarchy are not defini…
HIPO: Instruction Hierarchy via Constrained Reinforcement Learning
School of ECEE Arizona State University（亚利桑那州立大学电子与计算机工程学院）；Department of ECE The Ohio State University（俄亥俄州立大学电子工程系）；Computer Science Department University of Houston（休斯顿大学计算机科学系）；College of Engineering & Applied Science CU Boulder（科罗拉多大学博尔德分校工程与应用科学学院）；Dept. of Electrical Engineering & Computer Science United States Military Academy（美国军事学院电子工程与计算机科学系）
Human-AI Collaboration in Cloud Security: Cognitive Hierarchy-Driven Deep Reinforcement Learning
Rutgers University Electrical and Computer Engineering（新泽西罗格兰大学电气与计算机工程系）；Rutgers University WINLAB, Electrical and Computer Engineering（新泽西罗格兰大学WINLAB电气与计算机工程系）
Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
University of Michigan（密歇根大学）；UNC, Chapel Hill（北卡罗来纳大学教堂山分校）；Georgia Tech（佐治亚理工学院）；Apple（苹果公司）；Robert Bosch SRL（博世股份有限公司）；Babeş-Bolyai University（巴贝什-博耶亚大学）
