arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2002.00874cs.LGmath.OCstat.ML
收藏

光滑凸包络随机逼近的小样本分析

Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes

Georgia Institute of Technology(佐治亚理工学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校);IBM Research NY(IBM纽约研究院)

Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, Karthikeyan Shanmugam

摘要:

随机逼近(SA)是一种用于求解不动点方程的流行方法,其中信息被噪声破坏。本文研究了一类涉及压缩映射的SA问题,给出了在不同步长下的有限样本误差界。其思想是利用广义Moreau包络构造一个光滑的李雅普诺夫函数,并证明SA的迭代相对于该李雅普诺夫函数具有负漂移。本文的结果在强化学习(RL)中也是适用的。特别是,我们用它来建立第一个已知的收敛速度的V-跟踪算法的偏离政策TD学习。此外,我们还使用它来研究TD学习的策略设置,并恢复现有的国家的最先进的结果$Q$学习。重要的是,我们的建设结果在状态空间的大小上的收敛界的对数依赖。

英文摘要:

Stochastic Approximation (SA) is a popular approach for solving fixed-point equations where the information is corrupted by noise. In this paper, we consider an SA involving a contraction mapping with respect to an arbitrary norm, and show its finite-sample error bounds while using different stepsizes. The idea is to construct a smooth Lyapunov function using the generalized Moreau envelope, and show that the iterates of SA have negative drift with respect to that Lyapunov function. Our result is applicable in Reinforcement Learning (RL). In particular, we use it to establish the first-known convergence rate of the V-trace algorithm for off-policy TD-learning. Moreover, we also use it to study TD-learning in the on-policy setting, and recover the existing state-of-the-art results for $Q$-learning. Importantly, our construction results in only a logarithmic dependence of the convergence bound on the size of the state-space.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑