arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1206.3285cs.AIcs.LGcs.SYeess.SY
收藏

具有线性函数逼近和优先级扫描的Dyna风格规划

Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping

University of Alberta(阿尔伯塔大学)

Richard S. Sutton, Csaba Szepesvari, Alborz Geramifard, Michael P. Bowling

AI总结:

本文提出一种基于模型的Dyna风格规划方法,扩展至线性函数逼近,证明其收敛性,并引入线性Dyna的优先级扫描算法。

摘要:

我们考虑在在线设置中高效学习最优控制策略和价值函数的问题,其中状态空间很大,且必须在每次与世界交互后获得估计。本文开发了一种显式的基于模型的方法,将Dyna架构扩展到线性函数逼近。Dyna风格规划通过从世界模型生成想象经验,然后将无模型强化学习算法应用于想象的状态转移来进行。我们的主要结果是证明,在自然条件下,线性Dyna风格规划收敛到一个独立于生成分布的唯一解。在策略评估设置中,我们证明极限点是最小二乘(LSTD)解。我们的结果的一个含义是,优先级扫描可以合理地扩展到线性逼近情况,即回溯到前驱特征而不是前驱状态。我们介绍了两种线性Dyna的优先级扫描版本,并在Mountain Car和Boyan Chain问题上简要展示了它们的经验性能。

英文摘要:

We consider the problem of efficiently learning optimal control policies and value functions over large state spaces in an online setting in which estimates must be available after each interaction with the world. This paper develops an explicitly model-based approach extending the Dyna architecture to linear function approximation. Dynastyle planning proceeds by generating imaginary experience from the world model and then applying model-free reinforcement learning algorithms to the imagined state transitions. Our main results are to prove that linear Dyna-style planning converges to a unique solution independent of the generating distribution, under natural conditions. In the policy evaluation setting, we prove that the limit point is the least-squares (LSTD) solution. An implication of our results is that prioritized-sweeping can be soundly extended to the linear approximation case, backing up to preceding features rather than to preceding states. We introduce two versions of prioritized sweeping with linear Dyna and briefly illustrate their performance empirically on the Mountain Car and Boyan Chain problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑