In this study, we address the challenge of learning generalizable policies for compositional tasks defined by logical specifications. These tasks consist of multiple temporally extended sub-tasks. Due to the sub-task inter-dependencies and sparse reward issue in long-horizon tasks, existing reinforcement learning (RL) approaches, such as task-conditioned and goal-conditioned policies, continue to struggle with slow convergence and sub-optimal performance in generalizing to compositional tasks. To overcome these limitations, we introduce a new hierarchical RL framework that enhances the efficiency and optimality of task generalization. At the high level, we present an implicit planner specifically designed for generalizing compositional tasks. This planner selects the next sub-task and estimates the multi-step return for completing the remaining task to complete from the current state. It learns a latent transition model and performs planning in the latent space by using a graph neural network (GNN). Subsequently, the high-level planner's selected sub-task guides the low-level agent to effectively handle long-horizon tasks, while the multi-step return encourages the low-level policy to account for future sub-task dependencies, enhancing its optimality. We conduct comprehensive experiments to demonstrate the framework's advantages over previous methods in terms of both efficiency and optimality.
Integrating Legal and Logical Specifications in Perception, Prediction, and Planning for Automated Driving: A Survey of Methods
Department of Computer Science and Mathematics, Freie Universität Berlin（自由大学柏林计算机科学与数学系）；Continental Automotive（大陆汽车）；Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术系）
Logical Specifications-guided Dynamic Task Sampling for Reinforcement Learning Agents
Tufts University（塔夫茨大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Georgia Tech Research Institute（佐治亚理工学院研究所）；University of Colorado Boulder（科罗拉多大学博尔德分校）
Verifiable and Compositional Reinforcement Learning Systems
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（得克萨斯大学奥斯汀分校奥登计算工程与科学研究所）；Uppsala University（乌普萨拉大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
Hybrid Compositional Reasoning for Reactive Synthesis from Finite-Horizon Specifications
Rice University（莱斯大学）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）
