随机性有助于导航粗糙的景观：比较相位恢复问题中基于梯度下降的算法
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
摘要：
在本文中，我们研究如何基于梯度的算法，如梯度下降，（多遍）随机梯度下降，其持久的变种，和朗之万算法导航非凸损失景观，其中能够达到最佳的泛化误差在有限的样本复杂度。我们认为损失景观的高维相位恢复问题作为一个典型的高度非凸的例子。我们观察到，对于相位恢复，梯度下降的随机变量能够达到完美的泛化控制参数的区域，梯度下降算法不是。我们应用动态平均场论从统计物理分析这些算法的完整轨迹在其连续的时间限制，一个温暖的开始，并为大系统的大小。我们进一步揭示了景观和算法的一些有趣的属性，例如梯度下降可以从较少的信息初始化中获得更好的泛化性能。
英文摘要：
In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best generalization error at limited sample complexity. We consider the loss landscape of the high-dimensional phase retrieval problem as a prototypical highly non-convex example. We observe that for phase retrieval the stochastic variants of gradient descent are able to reach perfect generalization for regions of control parameters where the gradient descent algorithm is not. We apply dynamical mean-field theory from statistical physics to characterize analytically the full trajectories of these algorithms in their continuous-time limit, with a warm start, and for large system sizes. We further unveil several intriguing properties of the landscape and the algorithms such as that the gradient descent can obtain better generalization properties from less informed initializations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
