Optimization on Riemannian manifolds widely arises in eigenvalue computation, density functional theory, Bose-Einstein condensates, low rank nearest correlation, image registration, and signal processing, etc. We propose an adaptive regularized Newton method which approximates the original objective function by the second-order Taylor expansion in Euclidean space but keeps the Riemannian manifold constraints. The regularization term in the objective function of the subproblem enables us to establish a Cauchy-point like condition as the standard trust-region method for proving global convergence. The subproblem can be solved inexactly either by first-order methods or a modified Riemannian Newton method. In the later case, it can further take advantage of negative curvature directions. Both global convergence and superlinear local convergence are guaranteed under mild conditions. Extensive computational experiments and comparisons with other state-of-the-art methods indicate that the proposed algorithm is very promising.
In this paper we present a cubic regularized Newton's method to minimize a smooth function over a Riemannian manifold. The proposed algorithm is shown to reach a second-order $ε$-stationary point within $\mathcal{O}(1/ε^{\frac{3}{2}})$ iterations, under the condition that the pullbacks are locally Lipschitz continuous, a condition that is shown to be satisfied if the manifold is compact. Furthermore, we present a local superlinear convergence result under some additional conditions.
We propose a stochastic variance-reduced cubic regularized Newton algorithm to optimize the finite-sum problem over a Riemannian submanifold of the Euclidean space. The proposed algorithm requires a full gradient and Hessian update at the beginning of each epoch while it performs stochastic variance-reduced updates in the iterations within each epoch. The iteration complexity of $O(ε^{-3/2})$ to obtain an $(ε,\sqrtε)$-second-order stationary point, i.e., a point with the Riemannian gradient norm upper bounded by $ε$ and minimum eigenvalue of Riemannian Hessian lower bounded by $-\sqrtε$, is es…
Newton's method for finding an unconstrained minimizer for strictly convex functions, generally speaking, does not converge from any starting point. We introduce and study the damped regularized Newton's method (DRNM). It converges globally for any strictly convex function, which has a minimizer in $R^n$. Locally DRNM converges with a quadratic rate. We characterize the neighborhood of the minimizer, where the quadratic rate occurs. Based on it we estimate the number of DRNM's steps required for finding an $\varepsilon$- approximation for the minimizer.
A damped Newton's method to find a singularity of a vector field in Riemannian setting is presented with global convergence study. It is ensured that the sequence generated by the proposed method reduces to a sequence generated by the Riemannian version of the classical Newton's method after a finite number of iterations, consequently its convergence rate is superlinear/quadratic. Moreover, numerical experiments illustrate that the damped Newton's method has better performance than Newton's method in number of iteration and computational time.
Optimization problems with composite functions consist of an objective function which is the sum of a smooth and a (convex) nonsmooth term. This particular structure is exploited by the class of proximal gradient methods and some of their generalizations like proximal Newton and quasi-Newton methods. In this paper, we propose a regularized proximal quasi-Newton method whose main features are: (a) the method is globally convergent to stationary points, (b) the globalization is controlled by a regularization parameter, no line search is required, (c) the method can be implemented very efficientl…
An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes $\mathcal{O}(ε^{-3/2})$ iterations to drive the norm of the gradient of the objective function below a prescribed positive real number $ε$ and can take $\mathcal{O}(ε^{-3})$ iterations to drive the leftmost eigenvalue of the Hessian of the objective above $-ε$. The proposed algorithm is a general framework that covers a wide range of techniques including quadratically and cubically regularized Newton methods, such as the Adaptive Regularisation using Cubics (ARC) method and the recently p…
In this paper we study the Newton's method for finding a singularity of a differentiable vector field defined on a Riemannian manifold. Under the assumption of invertibility of covariant derivative of the vector field at its singularity, we establish the well definition of the method in a suitable neighborhood of this singularity. Moreover, we also show that the generated sequence by Newton method converges for the solution with superlinear rate.
Sparse logistic regression, as an effective tool of classification, has been developed tremendously in recent two decades, from its origination the $\ell_1$-regularized version to the sparsity constrained models. This paper is carried out on the sparsity constrained logistic regression by the Newton method. We begin with establishing its first-order optimality condition associated with a $τ$-stationary point. This point can be equivalently interpreted as a system of equations which is then efficiently solved by the Newton method. The method has a considerably low computational complexity and e…
We present a proximal quasi-Newton method in which the approximation of the Hessian has the special format of "identity minus rank one" (IMRO) in each iteration. The proposed structure enables us to effectively recover the proximal point. The algorithm is applied to $l_1$-regularized least square problem arising in many applications including sparse recovery in compressive sensing, machine learning and statistics. Our numerical experiment suggests that the proposed technique competes favourably with other state-of-the-art solvers for this class of problems. We also provide a complexity analysi…
As a tractable approach, regularization is frequently adopted in sparse optimization. This gives rise to the regularized optimization, aiming at minimizing the $\ell_0$ norm or its continuous surrogates that characterize the sparsity. From the continuity of surrogates to the discreteness of $\ell_0$ norm, the most challenging model is the $\ell_0$-regularized optimization. To conquer this hardness, there is a vast body of work on developing numerically effective methods. However, most of them only enjoy that either the (sub)sequence converges to a stationary point from the deterministic optimi…
This paper is concerned with an algorithm for finding a singularity of the nonsmooth vector fields. Firstly, we discuss the main results of the Newton method presented in [1] for solving the aforementioned problem. Combining this method with a nonmonotone line search strategy, we then propose a global version of the Newton Method. Finally, numerical experiments illustrating the practical advantages of the proposed scheme are reported.
The joint approximate diagonalization of non-commuting symmetric matrices is an important process in independent component analysis. This problem can be formulated as an optimization problem on the Stiefel manifold that can be solved using Riemannian optimization techniques. Among the available optimization techniques, this study utilizes the Riemannian Newton's method for the joint diagonalization problem on the Stiefel manifold, which has quadratic convergence. In particular, the resultant Newton's equation can be effectively solved by means of the Kronecker product and the vec and veck oper…
In this paper, we use Proximal Cubic regularized Newton Methods (PCNM) to optimize the sum of a smooth convex function and a non-smooth convex function, where we use inexact gradient and Hessian, and an inexact subsolver for the cubic regularized second-order subproblem. We propose inexact variants of PCNM and accelerated PCNM respectively, and show that both variants can achieve the same convergence rate as in the exact case, provided that the errors in the inexact gradient, Hessian and subsolver decrease at appropriate rates. Meanwhile, in the online stochastic setting where data comes endle…
In this paper, we propose a cubic regularized Newton (CRN) method for solving convex-concave saddle point problems (SPP). At each iteration, a cubic regularized saddle point subproblem is constructed and solved, which provides a search direction for the iterate. With properly chosen stepsizes, the method is shown to converge to the saddle point with global linear and local superlinear convergence rates, if the saddle point function is gradient Lipschitz and strongly-convex-strongly-concave. In the case that the function is merely convex-concave, we propose a homotopy continuation (or path-foll…
In this paper, we extend some results of nonsmooth analysis from Euclidean context to the Riemannian setting. In particular, we discuss the concept and some properties of locally Lipschitz continuous vector fields on Riemannian settings, such as Clarke generalized covariant derivative, upper semicontinuity and Rademacher theorem. We also present a version of Newton method for finding a singularity of a special class of locally Lipschitz continuous vector fields. Under mild conditions, we establish the well-definedness and local convergence of the sequence generated by the method in a neighborh…
A robust affine invariant version of Kantorovich's theorem on Newton's method, for finding a zero of a differentiable vector field defined on a complete Riemannian manifold, is presented in this paper. In the analysis presented, the classical Lipschitz condition is relaxed by using a general majorant function, which allow to establish existence and local uniqueness of the solution as well as unifying previously results pertaining Newton's method. The most important in our analysis is the robustness, namely, is given a prescribed ball, around the point satisfying Kantorovich's assumptions, ensu…
Matrix functions are utilized to rewrite smooth spectral constrained matrix optimization problems as smooth unconstrained problems over the set of symmetric matrices which are then solved via the cubic-regularized Newton method. A second-order chain rule identity for matrix functions is proven to compute the higher-order derivatives to implement cubic-regularized Newton, and a new convergence analysis is provided for cubic-regularized Newton for matrix vector spaces. We demonstrate the applicability of our approach by conducting numerical experiments on both synthetic and real datasets. In our…
Shape optimization based on the shape calculus is numerically mostly performed by means of steepest descent methods. This paper provides a novel framework to analyze shape-Newton optimization methods by exploiting a Riemannian perspective. A Riemannian shape Hessian is defined yielding often sought properties like symmetry and quadratic convergence for Newton optimization methods.
The goal of this paper is to study approaches to bridge the gap between first-order and second-order type methods for composite convex programs. Our key observations are: i) Many well-known operator splitting methods, such as forward-backward splitting (FBS) and Douglas-Rachford splitting (DRS), actually define a fixed-point mapping; ii) The optimal solutions of the composite convex program and the solutions of a system of nonlinear equations derived from the fixed-point mapping are equivalent. Solving this kind of system of nonlinear equations enables us to develop second-order type methods. …
Parameter-Free Cubic-Regularized Newton Method: Sharp Complexity and Generalized Smoothness
Graduate School of Information Science and Technology, University of Tokyo（东京大学信息理工学系研究科）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能研究中心）
We consider the minimization of non-convex quadratic forms regularized by a cubic term, which exhibit multiple saddle points and poor local minima. Nonetheless, we prove that, under mild assumptions, gradient descent approximates the $\textit{global minimum}$ to within $\varepsilon$ accuracy in $O(\varepsilon^{-1}\log(1/\varepsilon))$ steps for large $\varepsilon$ and $O(\log(1/\varepsilon))$ steps for small $\varepsilon$ (compared to a condition number we define), with at most logarithmic dependence on the problem dimension. When we use gradient descent to approximate the cubic-regularized Ne…
In this work, we generalized and unified recent two completely different works of Jascha \cite{sohl2014fast} and Lee \cite{lee2012proximal} respectively into one by proposing the \textbf{prox}imal s\textbf{to}chastic \textbf{N}ewton-type gradient (PROXTONE) method for optimizing the sums of two convex functions: one is the average of a huge number of smooth convex functions, and the other is a non-smooth convex function. While a set of recently proposed proximal stochastic gradient methods, include MISO, Prox-SDCA, Prox-SVRG, and SAG, converge at linear rates, the PROXTONE incorporates second …
We propose a stochastic variance-reduced cubic regularized Newton method for non-convex optimization. At the core of our algorithm is a novel semi-stochastic gradient along with a semi-stochastic Hessian, which are specifically designed for cubic regularization method. We show that our algorithm is guaranteed to converge to an $(ε,\sqrtε)$-approximately local minimum within $\tilde{O}(n^{4/5}/ε^{3/2})$ second-order oracle calls, which outperforms the state-of-the-art cubic regularization algorithms including subsampled cubic regularization. Our work also sheds light on the application of varia…
In this paper, we formulate the Load Flow (LF) problem in radial electricity distribution networks as an unconstrained Riemannian optimization problem, consisting of two manifolds, and we consider alternative retractions and initialization options. Our contribution is a novel LF solution method, which we show belongs to the family of Riemannian approximate Newton methods guaranteeing monotonic descent and local superlinear convergence rate. To the best of our knowledge, this is the first exact LF solution method employing Riemannian optimization. Extensive numerical comparisons on several test…
Ohio State University（俄亥俄州立大学）；Georgia Institute of Technology（佐治亚理工学院）
This note considers the inexact cubic-regularized Newton's method (CR), which has been shown in \cite{Cartis2011a} to achieve the same order-level convergence rate to a secondary stationary point as the exact CR \citep{Nesterov2006}. However, the inexactness condition in \cite{Cartis2011a} is not implementable due to its dependence on future iterates variable. This note fixes such an issue by proving the same convergence rate for nonconvex optimization under an inexact adaptive condition that depends on only the current iterate. Our proof controls the sufficient decrease of the function value …
