We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parameters and gradients, similar to the BFGS algorithm. Our approach involves two preconditioners: a matrix-free preconditioner and a low-rank approximation preconditioner. We update both preconditioners online using a criterion that is robust to stochastic gradient noise and does not require line search or damping. To preserve the corresponding symmetry or invariance, our preconditioners are constrained to certain connected Lie groups. The Lie group's equivariance property simplifies the preconditioner fitting process, while its invariance property eliminates the need for damping, which is commonly required in second-order optimizers. As a result, the learning rate for parameter updating and the step size for preconditioner fitting are naturally normalized, and their default values work well in most scenarios. Our proposed approach offers a promising direction for improving the convergence of SGD with low computational overhead. We demonstrate that Preconditioned SGD (PSGD) outperforms SoTA on Vision, NLP, and RL tasks across multiple modern deep-learning architectures. We have provided code for reproducing toy and large scale experiments in this paper.
GMEMS Technologies and Spectimbre（GMEMS技术和Spectimbre（公司））；Fairview Way, Milpitas, CA 95035（366 Fairview Way, Milpitas, CA 95035（地址，非机构））
We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and meth…
A matrix free and a low rank approximation preconditioner are proposed to accelerate the convergence of stochastic gradient descent (SGD) by exploiting curvature information sampled from Hessian-vector products or finite differences of parameters and gradients similar to the BFGS algorithm. Both preconditioners are fitted with an online updating manner minimizing a criterion that is free of line search and robust to stochastic gradient noise, and further constrained to be on certain connected Lie groups to preserve their corresponding symmetry or invariance, e.g., orientation of coordinates by…
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in significantly more complicated methods than SGD. This paper proposes a new method to estimate a preconditioner such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to Newton met…
Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
Department of Mathematics, Emory University（埃默里大学数学系）；Department of Mathematics, University of Minnesota Twin Cities（明尼苏达大学双城分校数学系）；Department of Computer Science, University of Minnesota Twin Cities（明尼苏达大学双城分校计算机科学系）；Department of Mathematics, University of Kentucky（肯塔基大学数学系）
MIT EECS（麻省理工学院电子工程与计算机科学系）；Google Research, New York（谷歌研究院纽约分部）
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the preconditioner is estimated in an online manner. By studying the preconditioner on its own, we elucidate its purpose: it rescales the stochastic gradient noise to be isotropic near stationary points, which helps escape saddle points. Furthermore, we show that adaptive methods can efficiently…
High-dimensional limit theorems for SGD: Momentum and Adaptive Step-sizes
Department of Statistics and Actuarial Science, University of Waterloo, Canada（统计与精算科学系，滑铁卢大学，加拿大）；Cheriton School of Computer Science, University of Waterloo, Canada（切尔顿计算机科学学院，滑铁卢大学，加拿大）
This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. We have improved the implementations of the original PSGD in several ways, e.g., new forms of preconditioners, more accurate Hessian vector product calculations, and better numerical stability with vanishing or ill-conditioned Hessian, etc.. We also have unrevealed the relationsh…
Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
Department of Mathematics, University of California, Berkeley, CA, USA（加州大学伯克利分校数学系）；Department of EECS, University of California, Berkeley, CA, USA（加州大学伯克利分校电子工程与计算机科学系）；Department of IEOR, University of California, Berkeley, CA, USA（加州大学伯克利分校工业工程与运营研究系）
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient descent (SGD). RNNs, especially the ones requiring extremely long term memories, are difficult to train. We have tested PSGD on a set of synthetic pathological RNN learning problems and the real world MNIST handwritten digit recognition task. Experimental results suggest that PSGD is …
Accurate estimates of network parameters are essential for modeling, monitoring, and control in power distribution systems. In this paper, we develop a physics-informed graphical learning algorithm to estimate network parameters of three-phase power distribution systems. Our proposed algorithm uses only readily available smart meter data to estimate the three-phase series resistance and reactance of the primary distribution line segments. We first develop a parametric physics-based model to replace the black-box deep neural networks in the conventional graphical neural network (GNN). Then we d…
Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differential equations and convergence analyses
School of Data Science and Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院和深圳大数据研究院，香港中文大学（深圳））
Learning to Control: The iUzawa-Net for Nonsmooth Optimal Control of Linear PDEs
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University（南洋理工大学数学科学系）；Department of Mathematics, The University of Hong Kong（香港大学数学系）；School of Mathematical Sciences, Nankai University（南开大学数学学院）
Texas Advanced Computing Center（德州高级计算中心）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Chicago（芝加哥大学）；Argonne National Laboratory（阿贡国家实验室）
Training neural networks with many processors can reduce time-to-solution; however, it is challenging to maintain convergence and efficiency at large scales. The Kronecker-factored Approximate Curvature (K-FAC) was recently proposed as an approximation of the Fisher Information Matrix that can be used in natural gradient optimizers. We investigate here a scalable K-FAC design and its applicability in convolutional neural network (CNN) training at scale. We study optimization techniques such as layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC …
Enhancing training of physics-informed neural networks using domain-decomposition based preconditioning strategies
Division of Applied Mathematics, Brown University（布朗大学应用数学系）；Euler Institute, Università della Svizzera italiana（瑞士意大利语区大学欧拉研究所）；UniDistance, Switzerland（瑞士远程大学）
College of Computer Science and Technology/Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院/人工智能学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）
Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove t…
Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective
Institution for Foundation of Data Science, Yale University（数据科学基础研究所，耶鲁大学）；Penn Institute for Computational Science, University of Pennsylvania（宾夕法尼亚大学计算科学研究所）；Department of Mathematics, City University of Hong Kong（香港城市大学数学系）；Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania（宾夕法尼亚大学机械工程与应用力学系）
Aalborg University（奥尔堡大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Implementations of SGD on distributed systems create new vulnerabilities, which can be identified and misused by one or more adversarial agents. Recently, it has been shown that well-known Byzantine-resilient gradient aggregation schemes are indeed vulnerable to informed attackers that can tailor the attacks (Fang et al., 2020; Xie et al., 2020b). We introduce MixTailor, a scheme based on randomization of the aggregation strategies that makes it impossible for the attacker to be fully informed. Deterministic schemes can be integrated into MixTailor on the fly without introducing any additional…
Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
本文受 Polyak-Ruppert 平均化启发，将平均化 Adam 用于 PDE、最优控制及图像分类的 DNN 训练，并在 PINN、deep BSDE 等多个数值实验中证明其优于标准 Adam 与 SGD。
