arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.16237cs.LG
收藏

通过梯度分裂缩小 SVRG 与 TD-SVRG 之间的差距

Closing the gap between SVRG and TD-SVRG with Gradient Splitting

Boston University(波士顿大学)

Arsenii Mustafin, Alex Olshevsky, Ioannis Ch. Paschalidis

AI总结:

本文利用梯度分裂视角简化并融合 TD 学习与 SVRG,在学习率 $1/8$ 下获得与凸优化 SVRG 相同的几何收敛界,并通过实验验证。

摘要:

时序差分(TD)学习是强化学习中的一种策略评估方法,其性能可通过方差缩减方法得到提升。近来,多项工作试图将 TD 学习与随机方差缩减梯度(SVRG)方法融合,以实现几何收敛速率。然而,所得收敛速率明显弱于 SVRG 在凸优化设定下所能达到的结果。在本工作中,我们利用近期对 TD 学习的一种解释,即将其视为对适当选择函数的梯度进行分裂,从而简化算法并将 TD 与 SVRG 融合。我们的主要结果是在预定学习率为 $1/8$ 时得到一个几何收敛界,该界与凸设定下 SVRG 可用的收敛界完全相同。我们的理论发现得到了一组实验的支持。

英文摘要:

Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have sought to fuse TD learning with Stochastic Variance Reduced Gradient (SVRG) method to achieve a geometric rate of convergence. However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization. In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG. Our main result is a geometric convergence bound with predetermined learning rate of $1/8$, which is identical to the convergence bound available for SVRG in the convex setting. Our theoretical findings are supported by a set of experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑