arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1709.04221math.OC
收藏

基于有效核梯度时间差分的连续MDP策略评估

Policy Evaluation in Continuous MDPs with Efficient Kernelized Gradient Temporal Difference

Alec Koppel, Garrett Warnell, Ethan Stump, Peter Stone, Alejandro Ribeiro

摘要:

我们考虑无限空间的无限时域折扣马尔可夫判定问题(MDP)的政策评估。我们重新制定这个任务的组合随机规划的函数值的决策变量,属于再生核希尔伯特空间(RKHS)。我们通过一个新的随机准梯度法操作与随机稀疏子空间投影的功能推广来解决这个问题。其结果是梯度时间差分学习的扩展,产生非线性参数化的Bellman评估方程的解的值函数估计。我们的主要贡献是一个内存有效的非参数随机方法,保证准确收敛到Bellman不动点的概率$1$与衰减步长。此外,在步长不变的情况下,我们得到了平均收敛到一个邻域,并且值函数估计具有有限的复杂性。在山车域中,我们观察到更快的收敛到更低的贝尔曼错误的解决方案比现有的方法所需的内存的一小部分。

英文摘要:

We consider policy evaluation in infinite-horizon discounted Markov decision problems (MDPs) with infinite spaces. We reformulate this task a compositional stochastic program with a function-valued decision variable that belongs to a reproducing kernel Hilbert space (RKHS). We approach this problem via a new functional generalization of stochastic quasi-gradient methods operating in tandem with stochastic sparse subspace projections. The result is an extension of gradient temporal difference learning that yields nonlinearly parameterized value function estimates of the solution to the Bellman evaluation equation. Our main contribution is a memory-efficient non-parametric stochastic method guaranteed to converge exactly to the Bellman fixed point with probability $1$ with attenuating step-sizes. Further, with constant step-sizes, we obtain mean convergence to a neighborhood and that the value function estimates have finite complexity. In the Mountain Car domain, we observe faster convergence to lower Bellman error solutions than existing approaches with a fraction of the required memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑