有偏随机逼近的多步李雅普诺夫方法
A Multistep Lyapunov Approach for Finite-Time Analysis of Biased Stochastic Approximation
摘要：
受强化学习中时间差分（TD-）和Q-学习算法广泛应用的启发，本文在对随机噪声序列的“类遍历”假设下，研究了一类有偏随机逼近（SA）过程.一个精心设计的多步李雅普诺夫函数，展望未来的几个更新，以适应随机扰动（梯度偏差的控制）的基础上，我们证明了一般的迭代收敛的结果，并使用它来获得非渐近界的情况下，恒定步长的均方误差。这种新颖的前瞻性的观点，使有限时间分析偏置SA算法在一个大家庭的随机扰动可能。为了与现有的贡献直接比较，我们还证明了这些界限，将它们应用到TD-和Q-学习与线性函数逼近，在实际的马尔可夫链观测模型。TD学习算法和Q学习算法的所得有限时间误差界是同类算法中的第一个，在这个意义上，它对于未修改的版本（即，不做任何修改的参数更新）使用甚至非线性函数逼近器;以及马尔可夫链ii）在一般的混合条件和iii）从任何初始分布，其中至少有一个必须违反现有的结果是适用的。
英文摘要：
Motivated by the widespread use of temporal-difference (TD-) and Q-learning algorithms in reinforcement learning, this paper studies a class of biased stochastic approximation (SA) procedures under a mild "ergodic-like" assumption on the underlying stochastic noise sequence. Building upon a carefully designed multistep Lyapunov function that looks ahead to several future updates to accommodate the stochastic perturbations (for control of the gradient bias), we prove a general result on the convergence of the iterates, and use it to derive non-asymptotic bounds on the mean-square error in the case of constant stepsizes. This novel looking-ahead viewpoint renders finite-time analysis of biased SA algorithms under a large family of stochastic perturbations possible. For direct comparison with existing contributions, we also demonstrate these bounds by applying them to TD- and Q-learning with linear function approximation, under the practical Markov chain observation model. The resultant finite-time error bound for both the TD- as well as the Q-learning algorithms is the first of its kind, in the sense that it holds i) for the unmodified versions (i.e., without making any modifications to the parameter updates) using even nonlinear function approximators; as well as for Markov chains ii) under general mixing conditions and iii) starting from any initial distribution, at least one of which has to be violated for existing results to be applicable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
