Goal-conditioned reinforcement learning endows an agent with a large variety of skills, but it often struggles to solve tasks that require more temporally extended reasoning. In this work, we propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. This high-level policy predicts intermediate states halfway to the goal using the value function as a reachability metric. We don't require the policy to reach these subgoals explicitly. Instead, we use them to define a prior policy, and incorporate this prior into a KL-constrained policy iteration scheme to speed up and regularize learning. Imagined subgoals are used during policy learning, but not during test time, where we only apply the learned policy. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.
Tampere University（坦佩雷大学）；Aalto University（阿尔托大学）；Huawei Helsinki Research Center（华为赫尔辛基研究中心）
Hierarchical reinforcement learning (HRL) proposes to solve difficult tasks by performing decision-making and control at successively higher levels of temporal abstraction. However, off-policy HRL often suffers from the problem of a non-stationary high-level policy since the low-level policy is constantly changing. In this paper, we propose a novel HRL approach for mitigating the non-stationarity by adversarially enforcing the high-level policy to generate subgoals compatible with the current instantiation of the low-level policy. In practice, the adversarial learning is implemented by trainin…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Pohang University of Science and Technology (POSTECH)（浦项科技大学）
Recently, graph-based planning algorithms have gained much attention to solve goal-conditioned reinforcement learning (RL) tasks: they provide a sequence of subgoals to reach the target-goal, and the agents learn to execute subgoal-conditioned policies. However, the sample-efficiency of such RL schemes still remains a challenge, particularly for long-horizon tasks. To address this issue, we present a simple yet effective self-imitation scheme which distills a subgoal-conditioned policy into the target-goal-conditioned policy. Our intuition here is that to reach a target-goal, an agent should p…
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Huazhong University of Science and Technology（华中科技大学）；Xi’an Jiaotong University（西安交通大学）；University of Electronic Science and Technology of China（电子科技大学）
Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur（计算机科学与工程系，印度理工学院Kharagpur分校）；Department of Mechanical Engineering, Indian Institute of Technology Kharagpur（机械工程系，印度理工学院Kharagpur分校）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informat…
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（首尔国立大学电气与计算机工程系）；Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea（全州大学汽车工程系）
While reinforcement learning provides an appealing formalism for learning individual skills, a general-purpose robotic system must be able to master an extensive repertoire of behaviors. Instead of learning a large collection of skills individually, can we instead enable a robot to propose and practice its own behaviors automatically, learning about the affordances and behaviors that it can perform in its environment, such that it can then repurpose this knowledge once a new task is commanded by the user? In this paper, we study this question in the context of self-supervised goal-conditioned …
Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency c…
Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of generality, these skills must handle raw sensory input such as images. In this paper, we propose an algorithm that acquires such general-purpose skills by combining unsupervised representation learning and reinforcement learning of goal-conditioned policies. Since the particular goals that might be required at test-time are not known in advance, the agent performs a self-supervised…
Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
IIT Kanpur（印度理工学院坎浦尔分校）；University of Maryland, College Park（马里兰大学学院公园分校）；U.S. Army Research Laboratory（美国陆军研究实验室）；University of Texas, Austin（德克萨斯大学奥斯汀分校）；Oracle（Oracle公司）；University of Central Florida（中央佛罗里达大学）；University of Bath（巴斯大学）
Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；East China Normal University（华东师范大学）；Tongji University（同济大学）
Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is large. Searching in a large goal space poses difficulty for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a $k$-step adjacent region of the current state using an adjacency constrain…
Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
Department of Computer Science, University of Toronto, Toronto, Canada（多伦多大学计算机科学系）；Vector Institute, Toronto, Canada（向量研究所）；University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校）；Harvard University, Cambridge, USA（哈佛大学）
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
University of Warwick（华威大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；University of Liverpool（利物浦大学）
InfoBot: Transfer and Exploration via the Information Bottleneck
Mila（米拉研究所）；University of Montreal（蒙特利尔大学）；McGill University（麦吉尔大学）；Deepmind（深度思维）；Google Brain（谷歌大脑）；University College London（伦敦大学学院）；University of California, Berkeley（加州大学伯克利分校）
Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Department of Electrical and Computer Engineering (ECE), Seoul National University（电子与计算机工程系，首尔国立大学）；Interdisciplinary Program in Artificial Intelligence (IPAI), Seoul National University（人工智能跨学科项目，首尔国立大学）；ASRI / INMC, Seoul National University（ASRI/INMC，首尔国立大学）
The history of learning for control has been an exciting back and forth between two broad classes of algorithms: planning and reinforcement learning. Planning algorithms effectively reason over long horizons, but assume access to a local policy and distance metric over collision-free paths. Reinforcement learning excels at learning policies and the relative values of states, but fails to plan over long horizons. Despite the successes of each method in various domains, tasks that require reasoning over long horizons with limited feedback and high-dimensional observations remain exceedingly chal…
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which g…
Achieve What You Imagined: Learning to Align Actions with Visual Plans
KTH（瑞典皇家理工学院）；Beihang University（北京航空航天大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；Great Bay University（大湾区大学）
University of Amsterdam（阿姆斯特丹大学）；Salesforce Research（Salesforce研究院）
In many real-world scenarios, an autonomous agent often encounters various tasks within a single complex environment. We propose to build a graph abstraction over the environment structure to accelerate the learning of these tasks. Here, nodes are important points of interest (pivotal states) and edges represent feasible traversals between them. Our approach has two stages. First, we jointly train a latent pivotal state model and a curiosity-driven goal-conditioned policy in a task-agnostic manner. Second, provided with the information from the world graph, a high-level Manager quickly finds s…
Open-World Reinforcement Learning over Long Short-Term Imagination
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能大模型关键实验室、人工智能研究院、上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究院、技术东院）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院、华东师范大学）
