One major limitation to the applicability of Reinforcement Learning (RL) to many practical domains is the large number of samples required to learn an optimal policy. To address this problem and improve learning efficiency, we consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. Each layer is an MDP representing a coarser model of the one immediately below in the hierarchy. In this work, we propose a novel form of Reward Shaping where the solution obtained at the abstract level is used to offer rewards to the more concrete MDP, in such a way that the abstract solution guides the learning in the more complex domain. In contrast with other works in Hierarchical RL, our technique has few requirements in the design of the abstract models and it is also tolerant to modeling errors, thus making the proposed approach practical. We formally analyze the relationship between the abstract models and the exploration heuristic induced in the lower-level domain. Moreover, we prove that the method guarantees optimal convergence and we demonstrate its effectiveness experimentally.
Kyushu University（九州大学）；Technical University of Munich（慕尼黑工业大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）；Singapore Management University（新加坡管理大学）；University of Alberta（阿尔伯塔大学）；The University of Tokyo（东京大学）；Tianjin University（天津大学）
Environment Shaping in Reinforcement Learning using State Abstraction
EPFL（洛桑联邦理工学院）；MPI-SWS（马克斯·普朗克软件系统研究所）
强化学习（RL）代理所面临的核心挑战之一是在具有稀疏和噪声反馈信号的大状态空间的环境中有效地学习（接近）最优策略。在实际应用中，具有额外领域知识的专家可以通过\emph{shaping the environment}帮助加速学习过程，即，使学习环境更加友好。文献中流行的范例是\emph{potential-based reward shaping}，其中环境的奖励函数使用潜在函数用额外的局部奖励来增强。然而，基于潜力的奖励塑造的适用性在以下设置中受到限制：（i）状态空间非常大，并且计算适当的潜力函数具有挑战性;（ii）反馈信号是嘈杂的，并且即使具有整形的奖励，代理也可能陷入局部最优;以及（iii）单独改变奖励是不够的，并且有效的整形需要改变动态。我们解决这些限制的潜在的成形方法，并提出了一个新的框架\emph{environment shaping using state abstraction}。我们的主要思想是压缩环境的大状态空间与嘈杂的信号到一个抽象的空间，并使用这种抽象在创建更平滑，更有效的反馈信号的代理。我们研究了我们的抽象为基础的环境塑造的理论基础，并表明，代理人的政策在塑造的环境中保存在原始环境中接近最优的行为。
EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Inria（法国国家信息与自动化研究所）；Université de Bordeaux（波尔多大学）；ISIR Sorbonne Université（索邦大学智能系统与机器人研究所）；Univ Angers（昂热大学）；Microsoft Research Montreal（微软研究院蒙特利尔分院）
One-Step Bellman Alignment Enables Provably Efficient Transfer in Online RL
Department of Technology, Operations, & Statistics, New York University（纽约大学技术、运营与统计系）；Department of Operations Research & Financial Engineering, Princeton University（普林斯顿大学运筹学与金融工程系）；Department of Computer Science, Dartmouth College（达特茅斯学院计算机科学系）
