马尔可夫链随机梯度法的稳定性和推广
Stability and Generalization for Markov Chain Stochastic Gradient Methods
摘要：
近年来，马尔可夫链随机梯度法（MC-SGMs）的研究取得了很大的进展，主要集中在求解极小化问题的收敛性分析上。在本文中，我们提供了一个全面的推广分析MC-SGMs的最小化和极大极小问题，通过镜头的算法稳定性的框架内的统计学习理论。对于经验风险最小化（ERM）问题，通过引入平均变元稳定性，建立了光滑和非光滑情形下的最优超额种群风险界.对于极大极小问题，我们建立了平均参数稳定性和推广误差之间的定量关系，推广了已有的一致稳定性\cite{lei2021stability}的结果。进一步给出了凸-凹问题在期望和高概率下的第一次近似最优收敛速度，结合我们的稳定性结果，证明了光滑和非光滑情形下的最优泛化界都可以得到.据我们所知，这是第一次泛化分析的梯度采样时，从马尔可夫过程的SGM。
英文摘要：
Recently there is a large amount of work devoted to the study of Markov chain stochastic gradient methods (MC-SGMs) which mainly focus on their convergence analysis for solving minimization problems. In this paper, we provide a comprehensive generalization analysis of MC-SGMs for both minimization and minimax problems through the lens of algorithmic stability in the framework of statistical learning theory. For empirical risk minimization (ERM) problems, we establish the optimal excess population risk bounds for both smooth and non-smooth cases by introducing on-average argument stability. For minimax problems, we develop a quantitative connection between on-average argument stability and generalization error which extends the existing results for uniform stability \cite{lei2021stability}. We further develop the first nearly optimal convergence rates for convex-concave problems both in expectation and with high probability, which, combined with our stability results, show that the optimal generalization bounds can be attained for both smooth and non-smooth cases. To the best of our knowledge, this is the first generalization analysis of SGMs when the gradients are sampled from a Markov process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
