Many machine learning models involve solving optimization problems. Thus, it is important to deal with a large-scale optimization problem in big data applications. Recently, subsampled Newton methods have emerged to attract much attention due to their efficiency at each iteration, rectified a weakness in the ordinary Newton method of suffering a high cost in each iteration while commanding a high convergence rate. Other efficient stochastic second order methods are also proposed. However, the convergence properties of these methods are still not well understood. There are also several important gaps between the current convergence theory and the performance in real applications. In this paper, we aim to fill these gaps. We propose a unifying framework to analyze both local and global convergence properties of second order methods. Based on this framework, we present our theoretical results which match the performance in real applications well.
We propose a new modification of Newton iteration for finding some nonnegative Z-eigenpairs of a nonnegative tensor. The method has local quadratic convergence to a nonnegative eigenpair of a nonnegative tensor, under the usual assumption guaranteeing the local quadratic convergence of the original Newton iteration.
A Multilevel Newton Iteration Method for Eigenvalue Problems
Memorial University of Newfoundland（纽芬兰纪念大学）；Tianjin University of Finance and Economics（天津财经大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
The nonlinear eigen-problem $ Ax+F(x)=λx$ is studied where $A$ is an $n\times n$ irreducible Stieltjes matrix. Under certain conditions, this problem has a unique positive solution. We show that, starting from a multiple of the positive eigenvector of $A$, the Newton-like iteration for this problem converges monotonically. Numerical results illustrate the effectiveness of this Newton-like method.
The cost of a partitioned fluid-structure interaction scheme is typically assessed by the number of coupling iterations required per time step, while ignoring the Newton loops within the nonlinear sub-solvers. In this work, we discuss why these single-field iterations deserve more attention when evaluating the coupling's efficiency and how to find the optimal number of Newton steps per coupling iteration.
A shift splitting modified Newton-type (SSMN) iteration method is introduced for solving large sparse generalized absolute value equations (GAVEs). The SSMN method is established by replacing the regularized splitting of the coefficient matrix of the linear part, which is employed in the modified Newton-type (MN) iteration method, with the shift splitting of the matrix. The conditions for the convergence of the proposed method are discussed in depth for the cases when the coefficient matrix is a general matrix, a symmetric positive definite matrix, and an $H_{+}$-matrix. Through two numerical …
Fast convergent, accurate, computationally efficient, parallelizable, and robust matrix inversion and parameter estimation algorithms are required in many time-critical and accuracy-critical applications such as system identification, signal and image processing, network and big data analysis, machine learning and in many others. This paper introduces new composite power series expansion with optionally chosen rates (which can be calculated simultaneously on parallel units with different computational capacities) for further convergence rate improvement of high order Newton-Schulz iteration. N…
There are thousands of papers on rootfinding for nonlinear scalar equations. Here is one more, to talk about an apparently new method, which I call ``Inverse Cubic Iteration'' (ICI) in analogy to the Inverse Quadratic Iteration in Richard Brent's zeroin method. The possibly new method is based on a cubic blend of tangent-line approximations for the inverse function. We rewrite this iteration for numerical stability as an average of two Newton steps and a secant step: only one new function evaluation and derivative evaluation is needed for each step. The total cost of the method is therefore on…
Systematic overview of Newton-Schulz and Durand iterations with convergence analysis and factorizations is presented in the chronological sequence in unified framework. Practical recommendations for the choice of the order and factorizations of the algorithms and integration into Richardson iteration are given. The simplest combination of Newton-Schulz and Richardson iteration is applied to the parameter estimation problem associated with the failure detection via evaluation of the frequency content of the signals in electrical network. The detection is performed on real data for which the sof…
Picard-Based Acceleration of Newton Continuation for Mean Field Game PDE Systems
New York University Shanghai（纽约大学上海）；NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai（纽约大学上海华东师范大学数学科学联合研究院）；Department of Mathematics, New York University Shanghai（纽约大学上海数学系）；Department of Data Science, New York University Shanghai（纽约大学上海数据科学系）；Department of Mathematics, New York University（纽约大学数学系）
This paper studies the performance Newton's iteration applied with Anderson acceleration for solving the incompressible steady Navier-Stokes equations. We manifest that this method converges superlinearly with a good initial guess, and moreover, a large Anderson depth decelerates the convergence speed comparing to a small Anderson depth. We observe that the numerical tests confirm these analytical convergence results, and in addition, Anderson acceleration sometimes enlarges the domain of convergence for Newton's method.
In this paper, we propose a simple sparse approximate inverse for triangular matrices (SAIT). Using the Jacobi iteration method, we obtain an expression of the exact inverse of triangular matrix, which is a finite series. The SAIT is constructed based on this series. We apply the SAIT matrices to iterative methods with ILU preconditioners. The two triangular solvers in the ILU preconditioning procedure are replaced by two matrix-vector multiplications, which can be fine-grained parallelized. We test this method by solving some linear systems and eigenvalue problems with preconditioned iterativ…
Previous papers have shown the impact of partial convergence of discretized PDE on the accuracy of tangent and adjoint linearizations. A series of papers suggested linearization of the fixed point iteration used in the solution process as a means of computing the sensitivities rather than linearizing the discretized PDE, as the lack of convergence of the nonlinear problem indicates that the discretized form of the governing equations has not been satisfied. These works showed that the accuracy of an approximate linearization depends in part on the convergence of the nonlinear system. This work…
Transform before linearizing: robust Newton methods for singular $p$-Laplace and $p$-Stokes equations
Courant Institute School of Mathematics, Computing and Data Science, New York University（纽约大学柯朗数学科学研究所）；ETSIT, Technical University of Madrid (UPM)（马德里理工大学ETSIT）
