We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, a procedure that is both costly and sensitive to noise. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained. We first validate our method, called CurveBall, on small problems with known closed-form solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers seem to struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. Code is available.
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；SAS Institute（SAS研究所）
In this work we describe an Adaptive Regularization using Cubics (ARC) method for large-scale nonconvex unconstrained optimization using Limited-memory Quasi-Newton (LQN) matrices. ARC methods are a relatively new family of optimization strategies that utilize a cubic-regularization (CR) term in place of trust-regions and line-searches. LQN methods offer a large-scale alternative to using explicit second-order information by taking identical inputs to those used by popular first-order methods such as stochastic gradient descent (SGD). Solving the CR subproblem exactly requires Newton's method,…
Purdue Univ.（普渡大学）；Univ. of Queensland（昆士兰大学）；ICSI（国际计算机科学研究所）；Univ. of California at Berkeley（加州大学伯克利分校）
First-order optimization methods, such as stochastic gradient descent (SGD) and its variants, are widely used in machine learning applications due to their simplicity and low per-iteration costs. However, they often require larger numbers of iterations, with associated communication costs in distributed environments. In contrast, Newton-type methods, while having higher per-iteration costs, typically require a significantly smaller number of iterations, which directly translates to reduced communication costs. In this paper, we present a novel distributed optimizer for classification problems,…
Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching
University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Sichuan University（四川大学）；Southwest Jiaotong University（西南交通大学）；Zhejiang University City College（浙江大学城市学院）
Support vector machines (SVMs) are successful modeling and prediction tools with a variety of applications. Previous work has demonstrated the superiority of the SVMs in dealing with the high dimensional, low sample size problems. However, the numerical difficulties of the SVMs will become severe with the increase of the sample size. Although there exist many solvers for the SVMs, only few of them are designed by exploiting the special structures of the SVMs. In this paper, we propose a highly efficient sparse semismooth Newton based augmented Lagrangian method for solving a large-scale convex…
Q-Newton: Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The University of Pittsburgh（匹兹堡大学）；The University of Chicago（芝加哥大学）；University of California Los Angeles（加州大学洛杉矶分校）
Recent advances in deep learning have allowed neural networks (NNs) to successfully replace traditional numerical solvers in many applications, thus enabling impressive computing gains. One such application is time domain simulation, which is indispensable for the design, analysis and operation of many engineering systems. Simulating dynamical systems with implicit Newton-based solvers is a computationally heavy task, as it requires the solution of a parameterized system of differential and algebraic equations at each time step. A variety of NN-based methodologies have been shown to successful…
A fast neural hybrid Newton solver adapted to implicit methods for nonlinear dynamics
organization= Department of Mathematics , addressline= The Hong Kong University of Science；organization= Mathematical Institute , addressline= University of Oxford , country= United Kingdom；organization= Laboratoire Jacques-Louis Lions (UMR 7598) , addressline= Sorbonne Université , country= France；Autonomous Driving Research Lab , addressline= HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute , city= Futian , state= Shenzhen , country= China
University of Queensland（昆士兰大学）；The Centre for Information Resilience（信息弹性中心）；International Computer Science Institute（国际计算机科学研究所）；Stanford University（斯坦福大学）；University of California at Berkeley（加州大学伯克利分校）
We consider a variant of inexact Newton Method, called Newton-MR, in which the least-squares sub-problems are solved approximately using Minimum Residual method. By construction, Newton-MR can be readily applied for unconstrained optimization of a class of non-convex problems known as invex, which subsumes convexity as a sub-class. For invex optimization, instead of the classical Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global convergence can be guaranteed under a weaker notion of joint regularity of Hessian and gradient. We also obtain Newton-MR's problem-independ…
Learning features from Newton's algorithm: a way to accelerate nonlinear parametrized PDE solvers
Centre Borelli（博雷利中心）；ENS Paris-Saclay（巴黎萨克雷高等师范学校）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；Michelin（米其林公司）；Centre de Recherche de Ladoux（拉杜研究中心）；BMBI（BMBI中心）；Université de Technologie de Compiègne（贡比涅技术大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems $Bx = b$ with positive definite $B$ for $x$. The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of $B$, which can be used to estimate errors. Recent probabilistic interpretations of the secant family of quasi-Newton optimization algorithms are extended. Combined with properties of the conjugate gradient algorithm, this leads to uncertainty-calibrated methods with very limited cost overhead o…
University of Edinburgh（爱丁堡大学）；University of Cambridge（剑桥大学）；Instituto de Telecomunicações（电信研究所）；Leiden University（莱顿大学）
Since its formulation by Sir Isaac Newton, the problem of solving the equations of motion for three bodies under their own gravitational force has remained practically unsolved. Currently, the solution for a given initialization can only be found by performing laborious iterative calculations that have unpredictable and potentially infinite computational cost, due to the system's chaotic nature. We show that an ensemble of solutions obtained using an arbitrarily precise numerical integrator can be used to train a deep artificial neural network (ANN) that, over a bounded time interval, provides…
PETScML: Second-order solvers for training regression problems in Scientific Machine Learning
Extreme Computing Research Center, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学极端计算研究中心）；Mathematical Institute, University of Oxford（牛津大学数学研究所）
