基于模型的强化学习的值等价原理
The Value Equivalence Principle for Model-Based Reinforcement Learning
摘要：
从数据中学习环境模型通常被视为构建智能强化学习（RL）代理的重要组成部分。通常的做法是将模型的学习与使用分开，通过构建一个正确预测观察到的状态转换的环境动态模型。在本文中，我们认为，有限的代表性资源的基于模型的RL代理更好地用于建立模型，是直接有用的基于价值的规划。作为我们的主要贡献，我们介绍了价值等价的原则：两个模型是价值等价的一组功能和政策，如果它们产生相同的贝尔曼更新。我们提出了一个公式化的模型学习问题的基础上的价值等价原则，并分析了如何设置的可行的解决方案是由政策和功能的选择的影响。具体来说，我们表明，当我们增加所考虑的政策和功能集时，价值等价模型的类会缩小，直到最终崩溃到一个与完美描述环境的模型相对应的点。在许多问题中，直接建模状态到状态的转换可能既困难又不必要。通过利用值等价原则，可以找到更简单的模型，而不会影响性能，节省计算和内存。我们通过实验将其与最大似然估计等更传统的模型进行比较，说明了值等效模型学习的好处。更一般地说，我们认为，价值等价的原则是RL中最近一些经验成功的基础，如价值迭代网络，Predictron，价值预测网络，TreeQN和MuZero，并为这些结果提供了第一个理论基础。
英文摘要：
Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources of model-based RL agents are better used to build models that are directly useful for value-based planning. As our main contribution, we introduce the principle of value equivalence: two models are value equivalent with respect to a set of functions and policies if they yield the same Bellman updates. We propose a formulation of the model learning problem based on the value equivalence principle and analyze how the set of feasible solutions is impacted by the choice of policies and functions. Specifically, we show that, as we augment the set of policies and functions considered, the class of value equivalent models shrinks, until eventually collapsing to a single point corresponding to a model that perfectly describes the environment. In many problems, directly modelling state-to-state transitions may be both difficult and unnecessary. By leveraging the value-equivalence principle one may find simpler models without compromising performance, saving computation and memory. We illustrate the benefits of value-equivalent model learning with experiments comparing it against more traditional counterparts like maximum likelihood estimation. More generally, we argue that the principle of value equivalence underlies a number of recent empirical successes in RL, such as Value Iteration Networks, the Predictron, Value Prediction Networks, TreeQN, and MuZero, and provides a first theoretical underpinning of those results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
