arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1110.4946math.OC
收藏

弱导数约束马尔可夫决策过程的实时强化学习

Real-Time Reinforcement Learning of Constrained Markov Decision Processes with Weak Derivatives

Vikram Krishnamurthy, Felisa Vazquez Abad

摘要:

我们提出了在线策略梯度算法计算的局部最优策略的约束,平均成本,有限状态马尔可夫决策过程。随机逼近算法需要估计成本函数相对于表征随机化策略的参数的梯度。我们提出了一个球坐标参数化,并提出了一种新的基于模拟的梯度估计方案,涉及弱导数(测量值微分)。与广泛使用的评分函数方法相比,这样的方法具有显著降低的方差。类似于神经动态规划算法(例如Q学习或时间差分法),本文提出的算法是基于模拟的,不需要明确的知识,如转移概率的基本参数。然而,不同于神经动态规划方法,这里提出的算法可以处理约束和时变参数。数值例子说明了算法的性能。这篇论文最初写于2004年。我们现在把它放在arxiv上的一个原因是,分数函数梯度估计器继续在在线强化学习文献中使用,即使它的方差随着给定的$n$数据点(对于马尔可夫过程)的$O(n)$而增长。相比之下,弱导数估计量具有显著更小的$O(1)$方差,如本文(和其他地方)所报道的。

英文摘要:

We present on-line policy gradient algorithms for computing the locally optimal policy of a constrained, average cost, finite state Markov Decision Process. The stochastic approximation algorithms require estimation of the gradient of the cost function with respect to the parameter that characterizes the randomized policy. We propose a spherical coordinate parametrization and present a novel simulation based gradient estimation scheme involving weak derivatives (measure-valued differentiation). Such methods have substantially reduced variance compared to the widely used score function method. Similar to neuro-dynamic programming algorithms (e.g. Q-learning or Temporal Difference methods), the algorithms proposed in this paper are simulation based and do not require explicit knowledge of the underlying parameters such as transition probabilities. However, unlike neuro-dynamic programming methods, the algorithms proposed here can handle constraints and time varying parameters. Numerical examples are given to illustrate the performance of the algorithms. This paper was originally written in 2004. One reason we are putting this on arxiv now is that the score function gradient estimator continues to be used in the online reinforcement learning literature even though its variance grows as $O(n)$ given $n$ data points (for a Markov process). In comparison the weak derivative estimator has significantly smaller variance of $O(1)$ as reported in this paper (and elsewhere).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑