We provide non-asymptotic convergence rates of the Polyak-Ruppert averaged stochastic gradient descent (SGD) to a normal random vector for a class of twice-differentiable test functions. A crucial intermediate step is proving a non-asymptotic martingale central limit theorem (CLT), i.e., establishing the rates of convergence of a multivariate martingale difference sequence to a normal random vector, which might be of independent interest. We obtain the explicit rates for the multivariate martingale CLT using a combination of Stein's method and Lindeberg's argument, which is then used in conjunction with a non-asymptotic analysis of averaged SGD proposed in [PJ92]. Our results have potentially interesting consequences for computing confidence intervals for parameter estimation with SGD and constructing hypothesis tests with SGD that are valid in a non-asymptotic sense.
A Dynamical Central Limit Theorem for Shallow Neural Networks
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Department of Chemistry, Stanford University（斯坦福大学化学系）；Center for Data Science, New York University（纽约大学数据科学中心）
CLT-Optimal Parameter Error Bounds for Linear System Identification
Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, California, USA（明希电气与计算机工程系，南加州大学，洛杉矶，加利福尼亚州，美国）
Phase Transition of Spectral Fluctuations in Large Gram Matrices with a Variance Profile: A Unified Framework for Sparse CLTs
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；Division of Mathematical Sciences, Nanyang Technological University（南洋理工大学数理学院）
Central Limit Theorems for Stochastic Gradient Descent Quantile Estimators
Department of Statistics, University of Chicago（芝加哥大学统计系）；Department of Statistics and Data Science, Washington University in St. Louis（圣路易斯华盛顿大学统计与数据科学系）
Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$
Computer Science Department, Carnegie Mellon University（卡内基梅隆大学计算机科学系）；Department of Statistics, University of California, Davis（加州大学戴维斯分校统计系）；Department of Statistics, University of Chicago（芝加哥大学统计系）
Asymptotic Analysis of Conditioned Stochastic Gradient Descent
École Polytechnique（巴黎综合理工学院）；Institut Polytechnique de Paris（巴黎理工学院）；ENSAI（法国国立统计与信息分析学院）；École Nationale de la Statistique et de l’Analyse de l’Information（法国国立统计与信息分析学院）
