压缩型随机逼近的集中性:加性噪声与乘性噪声
Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise
AI总结:
该研究针对压缩型随机逼近算法,在加性和乘性噪声场景下建立最大集中界,提出自举论证方法,给出尾分布特性与不可能性结果,并验证其在线性SA和强化学习中的适用性。
摘要:
本文针对在任意范数(例如ℓ_∞范数)下满足压缩算子的随机逼近(SA)算法生成的迭代序列,建立了最大集中界。我们考虑迭代序列可能无界的两种场景:带有界乘性噪声的SA以及带子高斯加性噪声的SA。我们的最大集中不等式表明,在加性噪声场景下,收敛误差服从子高斯尾分布;在乘性噪声场景下,收敛误差服从Weibull尾分布(其衰减速度快于多项式衰减,但可能慢于指数衰减)。此外,我们给出了一个不可能性结果,证明在乘性噪声下通常不可能得到次指数尾分布。为建立最大集中界,我们提出了一种新颖的自举论证方法,该方法包括对收敛误差的广义莫罗包络的修正版本的矩生成函数进行界估计,并构造指数上鞅以应用Ville最大不等式。我们在线性SA和强化学习场景中验证了理论结果的适用性。
英文摘要:
In this paper, we establish maximal concentration bounds for the iterates generated by a stochastic approximation (SA) algorithm under a contractive operator with respect to some arbitrary norm (for example, the $\ell_\infty$-norm). We consider two settings where the iterates are potentially unbounded: SA with bounded multiplicative noise and SA with sub-Gaussian additive noise. Our maximal concentration inequalities state that the convergence error has a sub-Gaussian tail in the additive noise setting and a Weibull tail (which is faster than polynomial decay but could be slower than exponential decay) in the multiplicative noise setting. In addition, we provide an impossibility result showing that it is generally impossible to have sub-exponential tails under multiplicative noise. To establish the maximal concentration bounds, we develop a novel bootstrapping argument that involves bounding the moment-generating function of a modified version of the generalized Moreau envelope of the convergence error and constructing an exponential supermartingale to enable using Ville's maximal inequality. We demonstrate the applicability of our theoretical results in the context of linear SA and reinforcement learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。