arXiv 2002.083292020-02cs.LGstat.ML

价值驱动的后见之明模型

Value-driven Hindsight Modelling

DeepMind（DeepMind（深度思维））

Arthur Guez, Fabio Viola, Théophane Weber, Lars Buesing, Steven Kapturowski, Doina Precup, David Silver, Nicolas Heess

备注：: 9 pages + reference + appendix. NeurIPS 2020 version

摘要：

值估计是强化学习（RL）范式的关键组成部分。如何有效地从数据中学习值预测器是RL社区研究的主要问题之一，不同的方法以不同的方式利用问题域中的结构。模型学习可以利用观测序列中丰富的过渡结构，但这种方法通常对奖励函数不敏感。相比之下，无模型方法直接利用来自未来的兴趣量，但接收到潜在的弱标量信号（对回报的估计）。我们开发了一种位于这两个极端之间的RL中的表示学习方法：我们建议以一种可以直接帮助价值预测的方式学习模型。为此，我们确定未来轨迹的哪些特征提供了有用的信息来预测相关的回报。这提供了与任务直接相关的易处理的预测目标，并且因此可以加速学习价值函数。这个想法可以理解为事后推理，关于未来观察的哪些方面可以帮助预测过去的价值。我们展示了如何这可以帮助显着，即使在简单的政策评估设置。然后，我们在具有挑战性的领域中大规模测试我们的方法，包括57个Atari 2600游戏。

英文摘要：

Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn value predictors from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future, but receive a potentially weak scalar signal (an estimate of the return). We develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end, we determine which features of the future trajectory provide useful information to predict the associated return. This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文