政策外强化学习通过利用过去的经验,有望实现决策政策的样本有效学习。然而,在离线RL设置中-提供固定的交互集合并且不允许进一步的交互-已经表明标准的非策略RL方法可能会显着表现不佳。最近提出的方法通常旨在通过限制学习的策略以保持接近给定的交互数据集来解决这一缺点。在这项工作中,我们仔细研究了BCQ的一个重要简化-离线RL的一种先验方法-它删除了启发式设计选择,并自然地限制提取的策略,使其完全保持在给定行为策略的支持范围内。重要的是,在他们原来的理论考虑相反,我们通过引入一种新的备份操作,期望最大Q学习(EMaQ),这是更密切相关的实际算法,从而得到这个简化的算法。具体来说,除了分布支持之外,EMaQ还明确考虑了样本数量和建议分布,使我们能够导出新的次优性边界,可以作为离线RL问题复杂性的新度量。在离线RL设置中-这项工作的主要焦点- EMaQ在D4 RL基准测试中匹配并优于现有技术。在在线RL设置中,我们证明了EMaQ与Soft Actor Critic具有竞争力。我们的实证研究结果的主要贡献是证明了仔细的生成模型设计对估计行为策略的重要性,以及离线RL问题复杂性的直观概念。由于其简单的解释和较少的移动部件,例如没有表示策略的显式函数近似器,EMaQ可以作为未来工作的强大而易于实现的基线。
英文摘要:
Off-policy reinforcement learning holds the promise of sample-efficient learning of decision-making policies by leveraging past experience. However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform. Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions. In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy. Importantly, in contrast to their original theoretical considerations, we derive this simplified algorithm through the introduction of a novel backup operator, Expected-Max Q-Learning (EMaQ), which is more closely related to the resulting practical algorithm. Specifically, in addition to the distribution support, EMaQ explicitly considers the number of samples and the proposal distribution, allowing us to derive new sub-optimality bounds which can serve as a novel measure of complexity for offline RL problems. In the offline RL setting -- the main focus of this work -- EMaQ matches and outperforms prior state-of-the-art in the D4RL benchmarks. In the online RL setting, we demonstrate that EMaQ is competitive with Soft Actor Critic. The key contributions of our empirical findings are demonstrating the importance of careful generative model design for estimating behavior policies, and an intuitive notion of complexity for offline RL problems. With its simple interpretation and fewer moving parts, such as no explicit function approximator representing the policy, EMaQ serves as a strong yet easy to implement baseline for future work.
PyCFRL: A Python library for counterfactually fair offline reinforcement learning via sequential data preprocessing
Department of Statistics, University of Michigan, USA(统计学系,密歇根大学);Department of Biostatistics, University of Michigan, USA(生物统计学系,密歇根大学);Department of Statistics, London School of Economics, UK(统计学系,伦敦经济学院);Department of Health Behavior and Health Equity, School of Public Health, University of Michigan, USA(健康行为与健康公平系,公共卫生学院,密歇根大学)
The In-Sample Softmax for Offline Reinforcement Learning
Huawei Noah’s Ark Lab(华为诺亚方舟实验室);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所(Amii));University of Oxford(牛津大学)
Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
The University of Hong Kong, Hong Kong SAR(香港大学);INFIFORCE Intelligent Technology Co., Ltd., Hangzhou, China(INFIFORCE智能技术有限公司);Tsinghua University, Beijing, China(清华大学);Carnegie Mellon University, Pittsburgh, PA 15213, USA(卡内基梅隆大学)
A Tale of Two Cities: Pessimism and Opportunism in Offline Dynamic Pricing
Department of Statistics, Florida State University(佛罗里达州立大学统计系);Department of Management Science, University of Miami(迈阿密大学管理科学系);Department of Decision Sciences, The George Washington University(乔治华盛顿大学决策科学系)
Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization
Xi’an Jiaotong University(西安交通大学);Westlake University(西湖大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Georgia Institute of Technology(佐治亚理工学院);National University of Singapore(新加坡国立大学);Aalto University School of Electrical Engineering, Aalto University(阿alto大学电气工程学院,阿alto大学)
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与运载学院);Department of Computer Science, The University of Hong Kong(香港大学计算机科学系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
Data 61, CSIRO, Australia(数据61,CSIRO,澳大利亚);School of Computing, Macquarie University, Australia(计算学院,麦考瑞大学,澳大利亚);Adobe Research, USA(Adobe研究,美国);School of Computer Science and Engineering, University of New South Wales, Australia(计算机科学与工程学院,新南威尔士大学,澳大利亚)