用平滑和随机梯度处理良性非凸性
Tackling benign nonconvexity with smoothing and stochastic gradients
摘要：
非凸优化问题在机器学习中普遍存在，特别是在深度学习中。虽然这种复杂的问题往往可以成功地优化在实践中使用随机梯度下降（SGD），理论分析不能充分解释这种成功。特别是，标准分析没有显示SGD在非凸函数上的全局收敛性，而是显示收敛到稳定点（也可以是局部极小值或鞍点）。我们确定了广泛的一类非凸函数，我们可以表明，扰动SGD（梯度下降扰动随机噪声-覆盖SGD作为一种特殊情况）收敛到一个全局最小值（或其附近），相比之下，梯度下降没有噪声，可以卡在远离全局解的局部最小值。例如，在非凸函数是相对接近凸状（强凸或PL）的功能，我们表明，SGD可以线性收敛到全局最优。
英文摘要：
Non-convex optimization problems are ubiquitous in machine learning, especially in Deep Learning. While such complex problems can often be successfully optimized in practice by using stochastic gradient descent (SGD), theoretical analysis cannot adequately explain this success. In particular, the standard analyses do not show global convergence of SGD on non-convex functions, and instead show convergence to stationary points (which can also be local minima or saddle points). We identify a broad class of nonconvex functions for which we can show that perturbed SGD (gradient descent perturbed by stochastic noise -- covering SGD as a special case) converges to a global minimum (or a neighborhood thereof), in contrast to gradient descent without noise that can get stuck in local minima far from a global solution. For example, on non-convex functions that are relatively close to a convex-like (strongly convex or PL) function we show that SGD can converge linearly to a global optimum.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
