Reinforcement learning algorithms struggle on tasks with complex hierarchical dependency structures. Humans and other intelligent agents do not waste time assessing the utility of every high-level action in existence, but instead only consider ones they deem possible in the first place. By focusing only on what is feasible, or "afforded", at the present moment, an agent can spend more time both evaluating the utility of and acting on what matters. To this end, we present Hierarchical Affordance Learning (HAL), a method that learns a model of hierarchical affordances in order to prune impossible subtasks for more effective learning. Existing works in hierarchical reinforcement learning provide agents with structural representations of subtasks but are not affordance-aware, and by grounding our definition of hierarchical affordances in the present state, our approach is more flexible than the multitude of approaches that ground their subtask dependencies in a symbolic history. While these logic-based methods often require complete knowledge of the subtask hierarchy, our approach is able to utilize incomplete and varying symbolic specifications. Furthermore, we demonstrate that relative to non-affordance-aware methods, HAL agents are better able to efficiently learn complex tasks, navigate environment stochasticity, and acquire diverse skills in the absence of extrinsic supervision -- all of which are hallmarks of human learning.
SAMP-HDRL: Segmented Allocation with Momentum-Adjusted Utility for Multi-agent Portfolio Management via Hierarchical Deep Reinforcement Learning
School of AI and Advanced Computing, XJTLU Entrepreneur College (Taicang), Xi’an Jiaotong-Liverpool University（人工智能与先进计算学院，XJTLU创业学院（太仓），西安交通大学利物浦大学）；Department of Management Science and Engineering, School of Economics and Management, Xinjiang University（管理科学与工程系，经济管理学院，新疆大学）
DISCO-TAB: A Hierarchical Reinforcement Learning Framework for Privacy-Preserving Synthesis of Complex Clinical Data
Computational Science Research Center, San Diego State University（圣地亚哥州立大学计算科学研究中心）；School of Nursing and Department of Computer Science, University of California, Irvine（加州大学尔湾分校护理学院与计算机科学系）；Fowler College of Business, San Diego State University（圣地亚哥州立大学福勒商学院）；Department of Computer Science, San Diego State University（圣地亚哥州立大学计算机科学系）
Improving Energy Efficiency in Femtocell Networks: A Hierarchical Reinforcement Learning Framework
VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；York-Zhejiang Lab for Cognitive Radio and Green Communications（约克-浙江认知无线电与绿色通信实验室）；Zhejiang University（浙江大学）
Hierarchical Utility Calibration for Structured Multiclass Decisions
The University of Osaka（大阪大学）；The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能项目中心（AIP））；Mila – Quebec AI Institute（米拉——魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Nagoya University（名古屋大学）
