This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established under some suitable conditions. We also prove that the semismoothness on submanifolds can be inherited from that in the ambient manifold. Finally, numerical experiments on compressed modes and (constrained) sparse principal component analysis illustrate the advantages of the proposed method.
On the role of semismoothness in nonsmooth numerical analysis: Theory
Johann Radon Institute for Computational and Applied Mathematics（约翰·拉德纳计算与应用数学研究所）；Institute of Information Theory and Automation, Czech Academy of Sciences（捷克科学院信息与自动化研究所）
On the role of semismoothness in the implicit programming approach to selected nonsmooth optimization problems
Institute of Information Theory and Automation, Czech Academy of Sciences（捷克科学院信息论与自动化研究所）；Johann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences（奥地利科学院约翰·拉登计算与应用数学研究所）；School of Mathematics, University of Birmingham（伯明翰大学数学学院）
In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and Hessian information of the smooth part of the objective function is available via calling stochastic first and second order oracles. The proposed method can be seen as a hybrid approach combining stochastic semismooth Newton steps and stochastic proximal gradient steps. Two inexact growth conditions are incorporated to monitor the convergence and the acceptan…
Support vector machine is an important and fundamental technique in machine learning. In this paper, we apply a semismooth Newton method to solve two typical SVM models: the L2-loss SVC model and the ε-L2-loss SVR model. The semismooth Newton method is widely used in optimization community. A common belief on the semismooth Newton method is its fast convergence rate as well as high computational complexity. Our contribution in this paper is that by exploring the sparse structure of the models, we significantly reduce the computational complexity, meanwhile keeping the quadratic convergence rat…
The paper starts with a concise description of the recently developed semismooth* Newton method for the solution of general inclusions. This method is then applied to a class of variational inequalities of the second kind. As a result, one obtains an implementable algorithm exhibiting a local superlinear convergence. Thereafter we suggest several globally convergent hybrid algorithms in which one combines the semismooth* Newton method with selected splitting algorithms for the solution of monotone variational inequalities. Their efficiency is documented by extensive numerical experiments.
We study a semismooth Newton-type method for the nearest doubly stochastic matrix problem where both differentiability and nonsingularity of the Jacobian can fail. The optimality conditions for this problem are formulated as a system of strongly semismooth functions. We show that the so-called local error bound condition does not hold for this system. Thus the guaranteed convergence rate of Newton-type methods is at most superlinear. By exploiting the problem structure, we construct a modified two step semismooth Newton method that guarantees a nonsingular Jacobian matrix at each iteration, an…
In the paper, a Newton-type method for the solution of generalized equations (GEs) is derived, where the linearization concerns both the single-valued and the multi-valued part of the considered GE. The method is based on the new notion of semismoothness${}^*$ which, together with a suitable regularity condition, ensure the local superlinear convergence. An implementable version of the new method is derived for a class of GEs, frequently arising in optimization and equilibrium models.
Augmented Lagrangian method (also called as method of multipliers) is an important and powerful optimization method for lots of smooth or nonsmooth variational problems in modern signal processing, imaging, optimal control and so on. However, one usually needs to solve the coupled and nonlinear system together and simultaneously, which is very challenging. In this paper, we proposed several semismooth Newton methods to solve the nonlinear subproblems arising in image restoration, which leads to several highly efficient and competitive algorithms for imaging processing. With the analysis of the…
Subgradient and Newton algorithms for nonsmooth optimization require generalized derivatives to satisfy subtle approximation properties: conservativity for the former and semismoothness for the latter. Though these two properties originate in entirely different contexts, we show that in the semi-algebraic setting they are equivalent. Both properties for a generalized derivative simply require it to coincide with the standard directional derivative on the tangent spaces of some partition of the domain into smooth manifolds. An appealing byproduct is a new short proof that semi-algebraic maps ar…
Subgradient methods comprise a fundamental class of nonsmooth optimization algorithms. Classical results show that certain subgradient methods converge sublinearly for general Lipschitz convex functions and converge linearly for convex functions that grow sharply away from solutions. Recent work has moreover extended these results to certain nonconvex problems. In this work we seek to improve the complexity of these algorithms, asking: is it possible to design a superlinearly convergent subgradient method? We provide a positive answer to this question for a broad class of sharp semismooth func…
The paper starts with a description of the SCD (subspace containing derivative) mappings and the SCD semismooth* Newton method for the solution of general inclusions. This method is then applied to a class of variational inequalities of the second kind. As a result, one obtains an implementable algorithm exhibiting a locally superlinear convergence. Thereafter we suggest several globally convergent hybrid algorithms in which one combines the SCD semismooth* Newton method with selected splitting algorithms for the solution of monotone variational inequalities. Finally we demonstrate the efficie…
This paper aims to develop a Newton-type method to solve a class of nonconvex composite programs. In particular, the nonsmooth part is possibly nonconvex. To tackle the nonconvexity, we develop a notion of strong prox-regularity which is related to the singleton property, Lipschitz continuity, and monotonicity of the associated proximal operator, and we verify it in various classes of functions, including weakly convex functions, indicator functions of proximally smooth sets, and two specific sphere-related nonconvex nonsmooth functions. In this case, the problem class we are concerned with co…
This work is concerned with the efficient optimization method for solving a large class of optimal mass transport problems. An inexact primal-dual algorithm is presented from the time discretization of a proper dynamical system, and by using the tool of Lyapunov function, the global (super-)linear convergence rate is established for function residual and feasibility violation. The proposed algorithm contains an inner problem that possesses strong semismoothness property and motivates the use of the semismooth Newton iteration. By exploring the hidden structure of the problem itself, the linear…
Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods to solve the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires few iterations to achi…
We consider the standard optimistic bilevel optimization problem, in particular upper- and lower-level constraints can be coupled. By means of the lower-level value function, the problem is transformed into a single-level optimization problem with a penalization of the value function constraint. For treating the latter problem, we develop a framework that does not rely on the direct computation of the lower-level value function or its derivatives. For each penalty parameter, the framework leads to a semismooth system of equations. This allows us to extend the semismooth Newton method to bileve…
We show that the Mordukhovich-stationarity system associated with a mathematical program with complementarity constraints (MPCC) can be equivalently written as a system of discontinuous equations which can be tackled with a semismooth Newton method. We show that the resulting algorithm can be interpreted as an active set strategy for MPCCs. Local fast convergence of the method is guaranteed under validity of an MPCC-tailored version of LICQ and a suitable second-order condition. In case of linear-quadratic MPCCs, the LICQ-type constraint qualification can be replaced by a weaker condition whic…
It is well known that Tikhonov regularization is one of the most commonly used methods for solving ill-posed problems. One of the most widely applied approaches is based on constructing a new dataset whose sample size is greater than the original one. The enlarged sample size may bring additional computational difficulties. In this paper, we aim to make full use of Tikhonov regularization and develop a dual based semismooth Newton (DSSN) method without destroying the structure of dataset. From the point of view of theory, we will show that \blue{the DSSN method is a globally convergent method …
We know that compressive sensing can establish stable sparse recovery results from highly undersampled data under a restricted isometry property condition. In reality, however, numerous problems are coherent, and vast majority conventional methods might work not so well. Recently, it was shown that using the difference between $\ell_1$- and $\ell_2$-norm as a regularization always has superior performance. In this paper, we propose an adaptive $\ell_p$-$\ell_{1-2}$ model where the $\ell_p$-norm with $p\geq 1$ measures the data fidelity and the $\ell_{1-2}$-term measures the sparsity. This prop…
We present a method to solve a special class of parameter identification problems for an elliptic optimal control problem to global optimality. The bilevel problem is reformulated via the optimal-value function of the lower-level problem. The reformulated problem is nonconvex and standard regularity conditions like Robinson's CQ are violated. Via a relaxation of the constraints, the problem can be decomposed into a family of convex problems and this is the basis for a solution algorithm. The convergence properties are analyzed. It is shown that a penalty method can be employed to solve this fa…
These lecture notes for a graduate course cover generalized derivative concepts useful in deriving necessary optimality conditions and numerical algorithms for nondifferentiable optimization problems in inverse problems, imaging, and PDE-constrained optimization. Treated are convex functions and subdifferentials, Fenchel duality, monotone operators and resolvents, Moreau--Yosida regularization, proximal point and (some) first-order splitting methods, Clarke subdifferentials, and semismooth Newton methods. The required background from functional analysis and calculus of variations is also brief…
This paper is concerned with the computation of the high-dimensional zero-norm penalized quantile regression estimator, defined as a global minimizer of the zero-norm penalized check loss function. To seek a desirable approximation to the estimator, we reformulate this NP-hard problem as an equivalent augmented Lipschitz optimization problem, and exploit its coupled structure to propose a multi-stage convex relaxation approach (MSCRA\_PPA), each step of which solves inexactly a weighted $\ell_1$-regularized check loss minimization problem with a proximal dual semismooth Newton method. Under a …
The octagonal shrinkage and clustering algorithm for regression (OSCAR), equipped with the $\ell_1$-norm and a pair-wise $\ell_{\infty}$-norm regularizer, is a useful tool for feature selection and grouping in high-dimensional data analysis. The computational challenge posed by OSCAR, for high dimensional and/or large sample size data, has not yet been well resolved due to the non-smoothness and inseparability of the regularizer involved. In this paper, we successfully resolve this numerical challenge by proposing a sparse semismooth Newton-based augmented Lagrangian method to solve the more g…
A Semismooth Newton-Based Proximal Augmented Lagrangian Method for Joint Estimation of Multiple Gaussian Graphical Models with Clustered Structure
School of Mathematics and Statistics, Fuzhou University（福州大学数学与统计学院）；Center for Applied Mathematics of Fujian Province, School of Mathematics and Statistics, Fuzhou University（福建省应用数学中心，福州大学数学与统计学院）
A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity
Institute of Applied Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院应用数学研究所）；School of Mathematical Sciences, University of Chinese Academy of Science（中国科学院大学数学科学学院）；School of Data Science, Fudan University（复旦大学数据科学系）
In this paper, we propose a new method that combines the inexact Newton method with a procedure to obtain a feasible inexact projection for solving constrained smooth and nonsmooth equations. The local convergence theorems are established under the assumption of smoothness or semismoothness of the function that defines the equation and its regularity at the solution. In particular, we show that a sequence generated by the method converges to a solution with linear, superlinear, or quadratic rate, under suitable conditions. Moreover, some numerical experiments are reported to illustrate the pra…
Support vector machine (SVM) has proved to be a successful approach for machine learning. Two typical SVM models are the L1-loss model for support vector classification (SVC) and $ε$-L1-loss model for support vector regression (SVR). Due to the nonsmoothness of the L1-loss function in the two models, most of the traditional approaches focus on solving the dual problem. In this paper, we propose an augmented Lagrangian method for the L1-loss model, which is designed to solve the primal problem. By tackling the nonsmooth term in the model with Moreau-Yosida regularization and the proximal operat…
Semismooth Newton methods for degenerate polyhedral projection
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Applied Mathematics, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院应用数学研究所）；Fudan University（复旦大学）
Powerful interior-point methods (IPM) based commercial solvers, such as Gurobi and Mosek, have been hugely successful in solving large-scale linear programming (LP) problems. The high efficiency of these solvers depends critically on the sparsity of the problem data and advanced matrix factorization techniques. For a large scale LP problem with data matrix $A$ that is dense (possibly structured) or whose corresponding normal matrix $AA^T$ has a dense Cholesky factor (even with re-ordering), these solvers may require excessive computational cost and/or extremely heavy memory usage in each inter…
Support Vector Machines (SVMs) are among the most popular and the best performing classification algorithms. Various approaches have been proposed to reduce the high computation and memory cost when training and predicting based on large-scale datasets with kernel SVMs. A popular one is the linearization framework, which successfully builds a bridge between the $L_1$-loss kernel SVM and the $L_1$-loss linear SVM. For linear SVMs, very recently, a semismooth Newton's method is proposed. It is shown to be very competitive and have low computational cost. Consequently, a natural question is wheth…
