Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to. In particular, it was shown that large initialization leads to the neural tangent kernel regime solution, whereas small initialization leads to so called "rich regimes". However, the initialization structure is richer than the overall scale alone and involves relative magnitudes of different weights and layers in the network. Here we show that these relative scales, which we refer to as initialization shape, play an important role in determining the learned model. We develop a novel technique for deriving the inductive bias of gradient-flow and use it to obtain closed-form implicit regularizers for multiple cases of interest.
Matrix factorization is a simple and natural test-bed to investigate the implicit regularization of gradient descent. Gunasekar et al. (2017) conjectured that Gradient Flow with infinitesimal initialization converges to the solution that minimizes the nuclear norm, but a series of recent papers argued that the language of norm minimization is not sufficient to give a full characterization for the implicit regularization. In this work, we provide theoretical and empirical evidence that for depth-2 matrix factorization, gradient flow with infinitesimal initialization is mathematically equivalent…
Most modern learning problems are highly overparameterized, meaning that there are many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (parameter vectors that perfectly interpolate the training data). Therefore, it is important to understand which interpolating solutions we converge to, how they depend on the initialization point and the learning algorithm, and whether they lead to different generalization performances. In this paper, we study these questions for the family of stochastic mirror descent (SMD) al…
Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrained mirror descent algorithms applied to the unregularized empirical risk with the squared loss for linear models and kernel methods. By completing an inequality that characterizes convexity for the squared loss, we identify an intrinsic link between offset Rademacher complexities and potential-based convergence analysis of mirror descent meth…
We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid saddle points.
This paper studies early-stopped mirror descent applied to noisy sparse phase retrieval, which is the problem of recovering a $k$-sparse signal $\mathbf{x}^\star\in\mathbb{R}^n$ from a set of quadratic Gaussian measurements corrupted by sub-exponential noise. We consider the (non-convex) unregularized empirical risk minimization problem and show that early-stopped mirror descent, when equipped with the hyperbolic entropy mirror map and proper initialization, achieves a nearly minimax-optimal rate of convergence, provided the sample size is at least of order $k^2$ (modulo logarithmic term) and …
Optimization Insights into Deep Diagonal Linear Networks
MaLGa, DIBRIS, Università degli Studi di Genova（基因瓦大学MaLGa、DIBRIS分校）；MaLGa, DIMA, Università degli Studi di Genova（基因瓦大学MaLGa、DIMA分校）；Center for Brains, Minds and Machines, MIT（MIT大脑、心智与机器中心）；Istituto Italiano di Tecnologia（意大利技术研究院）；Instituto de Alta investigaciòn (IAI), Universidad de Tarapacà（塔拉帕卡大学高级研究机构）
A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models
Center for Innovation in Data Engineering and Science (IDEAS) Department of Statistics and Data Science, the Wharton School University of Pennsylvania（创新数据工程与科学中心（IDEAS）统计与数据科学系，沃顿商学院，宾夕法尼亚大学）；Center for Innovation in Data Engineering and Science (IDEAS) University of Pennsylvania（创新数据工程与科学中心（IDEAS）宾夕法尼亚大学）；Center for Innovation in Data Engineering and Science (IDEAS) Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania（创新数据工程与科学中心（IDEAS）应用数学与计算科学研究生组，宾夕法尼亚大学）；Mathematical Institute for Data Science (MINDS) Department of Electrical and Computer Engineering Johns Hopkins University（数据科学数学研究所（MINDS）电气与计算机工程系，约翰霍普金斯大学）；Department of Electrical and Systems Engineering（电气与系统工程系；放射学系，宾夕法尼亚大学）；Department of Radiology University of Pennsylvania
Applying statistical learning theory to deep learning
Courant Institute of Mathematical Sciences（柯朗数学科学研究所）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Google Research（谷歌研究院）；Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；University of Pennsylvania（宾夕法尼亚大学）
