arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.07199cs.LG
收藏

关于梯度期望Sarsa($λ$)的收敛性

On Convergence of Gradient Expected Sarsa($λ$)

College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院)

Long Yang, Gang Zheng, Yu Zhang, Qian Zheng, Pengfei Li, Gang Pan

摘要:

研究了线性函数逼近下$\mathtt{Expected~Sarsa}(λ)$的收敛性。我们表明,应用离线估计(多步自举)$\mathtt{Expected~Sarsa}(λ)$是不稳定的离策略学习。在此基础上,基于凸凹鞍点框架,提出了一种收敛的$\mathtt{Gradient~Expected~Sarsa}(λ)$($\mathtt{GES}(λ)$)算法.理论分析表明,我们的$\mathtt{GES}(λ)$收敛到最优解的线性收敛速度,这是广泛的现有国家的最先进的梯度时间差学习算法。此外,我们还发展了一种李雅普诺夫函数技术来研究步长对$\mathtt{GES}(λ)$有限时间性能的影响,这种李雅普诺夫函数技术可以推广到其他GTD算法。最后,我们进行实验,以验证我们的$\mathtt{GES}(λ)$的有效性。

英文摘要:

We study the convergence of $\mathtt{Expected~Sarsa}(λ)$ with linear function approximation. We show that applying the off-line estimate (multi-step bootstrapping) to $\mathtt{Expected~Sarsa}(λ)$ is unstable for off-policy learning. Furthermore, based on convex-concave saddle-point framework, we propose a convergent $\mathtt{Gradient~Expected~Sarsa}(λ)$ ($\mathtt{GES}(λ)$) algorithm. The theoretical analysis shows that our $\mathtt{GES}(λ)$ converges to the optimal solution at a linear convergence rate, which is comparable to extensive existing state-of-the-art gradient temporal difference learning algorithms. Furthermore, we develop a Lyapunov function technique to investigate how the step-size influences finite-time performance of $\mathtt{GES}(λ)$, such technique of Lyapunov function can be potentially generalized to other GTD algorithms. Finally, we conduct experiments to verify the effectiveness of our $\mathtt{GES}(λ)$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑