arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.11091cs.LGstat.ML
收藏

EMaQ:用于简单而有效的离线和在线的期望最大Q学习算子RL

EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

University of Toronto(多伦多大学);Vector Institute(向量研究所)

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Shixiang Shane Gu

摘要:

政策外强化学习通过利用过去的经验,有望实现决策政策的样本有效学习。然而,在离线RL设置中-提供固定的交互集合并且不允许进一步的交互-已经表明标准的非策略RL方法可能会显着表现不佳。最近提出的方法通常旨在通过限制学习的策略以保持接近给定的交互数据集来解决这一缺点。在这项工作中,我们仔细研究了BCQ的一个重要简化-离线RL的一种先验方法-它删除了启发式设计选择,并自然地限制提取的策略,使其完全保持在给定行为策略的支持范围内。重要的是,在他们原来的理论考虑相反,我们通过引入一种新的备份操作,期望最大Q学习(EMaQ),这是更密切相关的实际算法,从而得到这个简化的算法。具体来说,除了分布支持之外,EMaQ还明确考虑了样本数量和建议分布,使我们能够导出新的次优性边界,可以作为离线RL问题复杂性的新度量。在离线RL设置中-这项工作的主要焦点- EMaQ在D4 RL基准测试中匹配并优于现有技术。在在线RL设置中,我们证明了EMaQ与Soft Actor Critic具有竞争力。我们的实证研究结果的主要贡献是证明了仔细的生成模型设计对估计行为策略的重要性,以及离线RL问题复杂性的直观概念。由于其简单的解释和较少的移动部件,例如没有表示策略的显式函数近似器,EMaQ可以作为未来工作的强大而易于实现的基线。

英文摘要:

Off-policy reinforcement learning holds the promise of sample-efficient learning of decision-making policies by leveraging past experience. However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform. Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions. In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy. Importantly, in contrast to their original theoretical considerations, we derive this simplified algorithm through the introduction of a novel backup operator, Expected-Max Q-Learning (EMaQ), which is more closely related to the resulting practical algorithm. Specifically, in addition to the distribution support, EMaQ explicitly considers the number of samples and the proposal distribution, allowing us to derive new sub-optimality bounds which can serve as a novel measure of complexity for offline RL problems. In the offline RL setting -- the main focus of this work -- EMaQ matches and outperforms prior state-of-the-art in the D4RL benchmarks. In the online RL setting, we demonstrate that EMaQ is competitive with Soft Actor Critic. The key contributions of our empirical findings are demonstrating the importance of careful generative model design for estimating behavior policies, and an intuitive notion of complexity for offline RL problems. With its simple interpretation and fewer moving parts, such as no explicit function approximator representing the policy, EMaQ serves as a strong yet easy to implement baseline for future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑