Developing agents to engage in complex goal-oriented dialogues is challenging partly because the main learning signals are very sparse in long conversations. In this paper, we propose a divide-and-conquer approach that discovers and exploits the hidden structure of the task to enable efficient policy learning. First, given successful example dialogues, we propose the Subgoal Discovery Network (SDN) to divide a complex goal-oriented task into a set of simpler subgoals in an unsupervised fashion. We then use these subgoals to learn a multi-level policy by hierarchical reinforcement learning. We demonstrate our method by building a dialogue agent for the composite task of travel planning. Experiments with simulated and real users show that our approach performs competitively against a state-of-the-art method that requires human-defined subgoals. Moreover, we show that the learned subgoals are often human comprehensible.
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Hierarchical reinforcement learning (HRL) holds great potential for sample-efficient learning on challenging long-horizon tasks. In particular, letting a higher level assign subgoals to a lower level has been shown to enable fast learning on difficult problems. However, such subgoal-based methods have been designed with static reinforcement learning environments in mind and consequently struggle with dynamic elements beyond the immediate control of the agent even though they are ubiquitous in real-world problems. In this paper, we introduce Hierarchical reinforcement learning with Timed Subgoa…
MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Southeast University-Monash University Joint Graduate School, Southeast University（东南大学-莫纳什大学联合研究生院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency c…
Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
This work is inspired by recent advances in hierarchical reinforcement learning (HRL) (Barto and Mahadevan 2003; Hengst 2010), and improvements in learning efficiency from heuristic-based subgoal selection, experience replay (Lin 1993; Andrychowicz et al. 2017), and task-based curriculum learning (Bengio et al. 2009; Zaremba and Sutskever 2014). We propose a new method to integrate HRL, experience replay and effective subgoal selection through an implicit curriculum design based on human expertise to support sample-efficient learning and enhance interpretability of the agent's behavior. Human …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informat…
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（首尔国立大学电气与计算机工程系）；Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea（全州大学汽车工程系）
We propose a novel hierarchical reinforcement learning framework for control with continuous state and action spaces. In our framework, the user specifies subgoal regions which are subsets of states; then, we (i) learn options that serve as transitions between these subgoal regions, and (ii) construct a high-level plan in the resulting abstract decision process (ADP). A key challenge is that the ADP may not be Markov, which we address by proposing two algorithms for planning in the ADP. Our first algorithm is conservative, allowing us to prove theoretical guarantees on its performance, which h…
Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
IIT Kanpur（印度理工学院坎浦尔分校）；University of Maryland, College Park（马里兰大学学院公园分校）；U.S. Army Research Laboratory（美国陆军研究实验室）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；Oracle（Oracle公司）；University of Central Florida（中央佛罗里达大学）；University of Bath（巴斯大学）
Hierarchical Reinforcement Learning with Targeted Causal Interventions
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland（计算机与通信科学系，瑞士洛桑联邦理工学院）；College of Management of Technology, EPFL, Lausanne, Switzerland（技术管理学院，瑞士洛桑联邦理工学院）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands（莱顿高级计算机科学研究所（LIACS），莱顿大学）
Imperial College London（帝国理工学院）；Laiye Network Technology Co. Ltd.（来也网络科技有限公司）；KAIST（韩国科学技术院）；University of Bath（巴斯大学）
Designing task-oriented dialogue systems is a challenging research topic, since it needs not only to generate utterances fulfilling user requests but also to guarantee the comprehensibility. Many previous works trained end-to-end (E2E) models with supervised learning (SL), however, the bias in annotated system utterances remains as a bottleneck. Reinforcement learning (RL) deals with the problem through using non-differentiable evaluation metrics (e.g., the success rate) as rewards. Nonetheless, existing works with RL showed that the comprehensibility of generated system utterances could be co…
Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea（计算机科学与工程系，成均馆大学，苏旺，韩国）；Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea（人工智能系，成均馆大学，苏旺，韩国）
Tampere University（坦佩雷大学）；Aalto University（阿尔托大学）；Huawei Helsinki Research Center（华为赫尔辛基研究中心）
Hierarchical reinforcement learning (HRL) proposes to solve difficult tasks by performing decision-making and control at successively higher levels of temporal abstraction. However, off-policy HRL often suffers from the problem of a non-stationary high-level policy since the low-level policy is constantly changing. In this paper, we propose a novel HRL approach for mitigating the non-stationarity by adversarially enforcing the high-level policy to generate subgoals compatible with the current instantiation of the low-level policy. In practice, the adversarial learning is implemented by trainin…
大多数Meta强化学习（meta-RL）方法通过直接优化原始动作空间上的策略参数来学习适应新任务。这样的算法在具有相对微小差异的任务中工作得很好。然而，当任务分布变得更宽时，直接学习这样的元策略将是非常低效的。在本文中，我们提出了一个新的元强化学习算法称为Meta Goal-generation for Hierarchical RL（MGHRL）。MGHRL不是直接在原始动作空间上为新任务生成策略，而是根据过去的经验在子目标上生成高级Meta策略，并将如何实现子目标的其余部分作为独立的RL子任务。我们在几个具有挑战性的模拟机器人环境中的经验结果表明，我们的方法可以从过去的经验中进行更有效和更广泛的元学习。
Heinrich Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学）；University of Cambridge（剑桥大学）
The dialogue management component of a task-oriented dialogue system is typically optimised via reinforcement learning (RL). Optimisation via RL is highly susceptible to sample inefficiency and instability. The hierarchical approach called Feudal Dialogue Management takes a step towards more efficient learning by decomposing the action space. However, it still suffers from instability due to the reward only being provided at the end of the dialogue. We propose the usage of an intrinsic reward based on information gain to address this issue. Our proposed reward favours actions that resolve unce…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Cambricon Technologies（寒武纪科技）；Southern University of Science and Technology（南方科技大学）；Institute of Software, CAS（中国科学院软件研究所）；University of Science and Technology of China（中国科学技术大学）
Hierarchical reinforcement learning (HRL) effectively improves agents' exploration efficiency on tasks with sparse reward, with the guide of high-quality hierarchical structures (e.g., subgoals or options). However, how to automatically discover high-quality hierarchical structures is still a great challenge. Previous HRL methods can hardly discover the hierarchical structures in complex environments due to the low exploration efficiency by exploiting the randomness-driven exploration paradigm. To address this issue, we propose CDHRL, a causality-driven hierarchical reinforcement learning fram…
Hohai University（河海大学）；The Hong Kong Polytechnic University（香港理工大学）
Designing an intelligent volume-weighted average price (VWAP) strategy is a critical concern for brokers, since traditional rule-based strategies are relatively static that cannot achieve a lower transaction cost in a dynamic market. Many studies have tried to minimize the cost via reinforcement learning, but there are bottlenecks in improvement, especially for long-duration strategies such as the VWAP strategy. To address this issue, we propose a deep learning and hierarchical reinforcement learning jointed architecture termed Macro-Meta-Micro Trader (M3T) to capture market patterns and execu…
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Huazhong University of Science and Technology（华中科技大学）；Xi’an Jiaotong University（西安交通大学）；University of Electronic Science and Technology of China（电子科技大学）
Imagine to Ensure Safety in Hierarchical Reinforcement Learning
Cognitive AI Systems Lab（认知人工智能系统实验室）；Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究所）；FRC Computer Science RAS（俄罗斯科学院联邦研究中心计算机科学研究所）
Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is large. Searching in a large goal space poses difficulty for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency constrain…
