This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function F , defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; an other choice is the output of the celebrated backpropagation algorithm, which is popular amongst practionners, and whose properties have recently been studied by Bolte and Pauwels [7]. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential BF of the mean function, it has been assumed in the literature that an oracle of BF is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of the differential inclusion. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function F .
Convergence rates for the stochastic gradient descent method for non-convex objective functions
University of Oxford（牛津大学）；Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学与自然科学研究所）；Universität Bielefeld（比勒费尔德大学）；ETH Zurich（苏黎世联邦理工学院）
Fastest Rates for Stochastic Mirror Descent Methods
相对平滑度-由Birnbaum et al.（2011）引入并由Bauschke et al.（2016）和Lu et al.（2016）重新发现的概念-概括了通常用于梯度类型方法分析的平滑度的标准概念。在这项工作中，我们正在采取的想法，从研究领域的随机凸优化，并使用它们，以获得更快的算法，最小化相对光滑函数。我们提出并分析了两种新算法：相对随机坐标下降（relRCD）和相对随机梯度下降（relSGD），两者都是标准平滑设置中著名算法的推广。我们提出的方法实际上可以看作是随机镜像下降算法的一个特殊实例。其中，relRCD对应于镜像下降算法的第一随机变体，具有线性收敛速率。
