Lambda策略迭代的性能界及其在俄罗斯方块游戏中的应用
Performance Bounds for Lambda Policy Iteration and Application to the Game of Tetris
摘要：
我们考虑离散时间无限时域最优控制问题的形式化马尔可夫决策过程。我们回顾了Bertsekas和Ioffe的工作，他们引入了$λ$策略迭代，这是一个由$λ$参数化的算法族，它概括了标准算法值迭代和策略迭代，并且与Sutton和Barto描述的时间差异算法TD（$λ$）有一些深层联系。我们深化了原来的理论发展的作者提供的收敛速度界，推广标准界的值迭代描述，例如由Puterman。然后，本文的主要贡献是发展的理论，这种算法时，它是在一个近似的形式，并表明这是健全的。这样做，我们扩展并统一了Munos为近似值迭代和近似策略迭代开发的单独分析。最后，我们重新审视使用该算法在训练的俄罗斯方块播放控制器最初所做的Bertsekas和Ioffe。我们提供了一个原始的性能界，可以应用到这样一个未折现的控制问题。我们的实证结果不同于Bertsekas和Ioffe的结果（他们最初被定性为“矛盾的”和“有趣的”），而更符合人们对学习实验的期望。我们讨论了这种差异的可能原因。
英文摘要：
We consider the discrete-time infinite-horizon optimal control problem formalized by Markov Decision Processes. We revisit the work of Bertsekas and Ioffe, that introduced $λ$ Policy Iteration, a family of algorithms parameterized by $λ$ that generalizes the standard algorithms Value Iteration and Policy Iteration, and has some deep connections with the Temporal Differences algorithm TD($λ$) described by Sutton and Barto. We deepen the original theory developped by the authors by providing convergence rate bounds which generalize standard bounds for Value Iteration described for instance by Puterman. Then, the main contribution of this paper is to develop the theory of this algorithm when it is used in an approximate form and show that this is sound. Doing so, we extend and unify the separate analyses developped by Munos for Approximate Value Iteration and Approximate Policy Iteration. Eventually, we revisit the use of this algorithm in the training of a Tetris playing controller as originally done by Bertsekas and Ioffe. We provide an original performance bound that can be applied to such an undiscounted control problem. Our empirical results are different from those of Bertsekas and Ioffe (which were originally qualified as "paradoxical" and "intriguing"), and much more conform to what one would expect from a learning experiment. We discuss the possible reason for such a difference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
