arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1212.2471cs.LGcs.AIcs.NAmath.NA
收藏

蒙特卡洛矩阵求逆策略评估

Monte Carlo Matrix Inversion Policy Evaluation

University of Waterloo(滑铁卢大学)

Fletcher Lu, Dale Schuurmans

AI总结:

提出使用蒙特卡洛矩阵求逆(MCMI)进行强化学习策略评估,通过重要性采样降低方差,并在运行时间和准确性上优于最大似然模型和时序差分方法。

摘要:

1950年,Forsythe和Leibler(1950)引入了一种统计技术,通过将矩阵逆的元素表征为一系列随机游走的期望值来求矩阵的逆。Barto和Duff(1994)随后展示了该技术与标准动态规划和时序差分方法之间的关系。蒙特卡洛矩阵求逆(MCMI)方法的优势在于,它相对于其他技术,在状态空间大小方面具有更好的可扩展性。在本文中,我们介绍了一种使用MCMI进行强化学习策略评估的算法。我们证明,MCMI在运行时间上优于基于最大似然模型的策略评估方法,并且在运行时间和准确性上都优于时序差分(TD)策略评估方法。我们进一步通过向算法添加重要性采样技术来降低估计器的方差,从而改进了MCMI策略评估。最后,我们展示了将MCMI扩展到大规模状态空间以进行策略改进的技术。

英文摘要:

In 1950, Forsythe and Leibler (1950) introduced a statistical technique for finding the inverse of a matrix by characterizing the elements of the matrix inverse as expected values of a sequence of random walks. Barto and Duff (1994) subsequently showed relations between this technique and standard dynamic programming and temporal differencing methods. The advantage of the Monte Carlo matrix inversion (MCMI) approach is that it scales better with respect to state-space size than alternative techniques. In this paper, we introduce an algorithm for performing reinforcement learning policy evaluation using MCMI. We demonstrate that MCMI improves on runtime over a maximum likelihood model-based policy evaluation approach and on both runtime and accuracy over the temporal differencing (TD) policy evaluation approach. We further improve on MCMI policy evaluation by adding an importance sampling technique to our algorithm to reduce the variance of our estimator. Lastly, we illustrate techniques for scaling up MCMI to large state spaces in order to perform policy improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑