arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.06828cs.LG
收藏

关于线性函数逼近下SARSA收敛性的研究

On the Convergence of SARSA with Linear Function Approximation

University of Virginia(弗吉尼亚大学);AlpacaML

Shangtong Zhang, Remi Tachet, Romain Laroche

AI总结:

本文研究了线性函数逼近下SARSA算法的收敛性,证明了投影SARSA收敛到有界区域的速率,且该区域远小于投影区域,适用于任意Lipschitz常数,填补了现有理论的空白。

摘要:

SARSA是一种经典的强化学习同策略控制算法,当与线性函数逼近结合时,已知会出现“喋喋不休”现象:SARSA不会发散,但会在一个有界区域内振荡。然而,关于SARSA收敛到该区域的速度以及该区域的大小,目前知之甚少。在本文中,我们通过展示投影SARSA收敛到一个有界区域的收敛速率,朝着解决这一开放问题取得了进展。重要的是,当奖励的幅度不太大时,该区域远小于我们投影所进入的区域。现有关于线性SARSA收敛到不动点的工作均要求SARSA策略改进算子的Lipschitz常数足够小;而我们的分析则适用于任意的Lipschitz常数,从而刻画了线性SARSA在新机制下的行为。

英文摘要:

SARSA, a classical on-policy control algorithm for reinforcement learning, is known to chatter when combined with linear function approximation: SARSA does not diverge but oscillates in a bounded region. However, little is known about how fast SARSA converges to that region and how large the region is. In this paper, we make progress towards this open problem by showing the convergence rate of projected SARSA to a bounded region. Importantly, the region is much smaller than the region that we project into, provided that the magnitude of the reward is not too large. Existing works regarding the convergence of linear SARSA to a fixed point all require the Lipschitz constant of SARSA's policy improvement operator to be sufficiently small; our analysis instead applies to arbitrary Lipschitz constants and thus characterizes the behavior of linear SARSA for a new regime.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑