This paper aims at developing two versions of the generalized Newton method to compute not merely arbitrary local minimizers of nonsmooth optimization problems but just those, which possess an important stability property known as tilt stability. We start with unconstrained minimization of continuously differentiable cost functions having Lipschitzian gradients and suggest two second-order algorithms of the Newton type: one involving coderivatives of Lipschitzian gradient mappings, and the other based on graphical derivatives of the latter. Then we proceed with the propagation of these algorithms to minimization of extended-real-valued prox-regular functions, while covering in this way problems of constrained optimization, by using Moreau envelops. Employing advanced techniques of second-order variational analysis and characterizations of tilt stability allows us to establish the solvability of subproblems in both algorithms and to prove the Q-superlinear convergence of their iterations.
The popular BFGS quasi-Newton minimization algorithm under reasonable conditions converges globally on smooth convex functions. This result was proved by Powell in 1976: we consider its implications for functions that are not smooth. In particular, an analogous convergence result holds for functions, like the Euclidean norm, that are nonsmooth at the minimizer.
We propose techniques for approximating bilevel optimization problems with non-smooth lower level problems that can have a non-unique solution. To this end, we substitute the expression of a minimizer of the lower level minimization problem with an iterative algorithm that is guaranteed to converge to a minimizer of the problem. Using suitable non-linear proximal distance functions, the update mappings of such an iterative algorithm can be differentiable, notwithstanding the fact that the minimization problem is non-smooth.
We consider a nonsmooth optimization problem on Riemannian manifold, whose objective function is the sum of a differentiable component and a nonsmooth convex function. We propose a manifold inexact augmented Lagrangian method (MIALM) for the considered problem. The problem is reformulated to a separable form. By utilizing the Moreau envelope, we get a smoothing subproblem at each iteration of the proposed method. Theoretically, under suitable assumptions, the convergence to critical point of the proposed method is established. In particular, under the condition of that the approximate global m…
A regularized minimization model with $l_1$-norm penalty (RP) is introduced for training the autoencoders that belong to a class of two-layer neural networks. We show that the RP can act as an exact penalty model which shares the same global minimizers, local minimizers, and d(irectional)-stationary points with the original regularized model under mild conditions. We construct a bounded box region that contains at least one global minimizer of the RP, and propose a linearly constrained regularized minimization model with $l_1$-norm penalty (LRP) for training autoencoders. A smoothing proximal …
This paper is devoted to the study of tilt stability of local minimizers, which plays an important role in both theoretical and numerical aspects of optimization. This notion has been comprehensively investigated in the unconstrained framework as well as for problems of nonlinear programming with $C^2$-smooth data. Available results for nonpolyhedral conic programs were obtained only under strong constraint nondegeneracy assumptions. Here we develop an approach of second-order variational analysis, which allows us to establish complete neighborhood and pointbased characterizations of tilt stab…
The main goal of this paper is developing the method of discrete approximations to derive necessary optimality conditions for a class of constrained sweeping processes with nonsmooth perturbations. Optimal control problems for sweeping processes have been recently recognized among the most interesting and challenging problems in modern control theory for discontinuous differential inclusions with irregular dynamics and implicit state constrained, while deriving necessary optimality conditions for their local minimizers have been significantly based on the smoothness of controlled dynamic pertu…
We introduce the concept of strong high-order approximate minimizers for nonconvex optimization problems. These apply in both standard smooth and composite non-smooth settings, and additionally allow convex or inexpensive constraints. An adaptive regularization algorithm is then proposed to find such approximate minimizers. Under suitable Lipschitz continuity assumptions, whenever the feasible set is convex, it is shown that using a model of degree $p$, this algorithm will find a strong approximate q-th-order minimizer in at most ${\cal O}\left(\max_{1\leq j\leq q}ε_j^{-(p+1)/(p-j+1)}\right)$ …
In this paper, we first study nonsmooth steepest descent method for nonsmooth functions defined on Hilbert space and establish the corresponding algorithm by proximal subgradients. Then, we use this algorithm to find stationary points for those functions satisfying prox-regularity and Lipschitzian continuity. As one application, the established algorithm is used to search the minimizer of lower semicontinuous and convex functions on finite-dimensional space. The convergent theorem, as one extension and improvement of the existing converging result for twice continuously differentiable convex f…
Beck and Teboulle's FISTA method for finding a minimizer of the sum of two convex functions, one of which has a Lipschitz continuous gradient whereas the other may be nonsmooth, is arguably the most important optimization algorithm of the past decade. While research activity on FISTA has exploded ever since, the mathematically challenging case when the original optimization problem has no minimizer has found only limited attention. In this work, we systematically study FISTA and its variants. We present general results that are applicable, regardless of the existence of minimizers.
Optimization problems with uncertainty in the constraints occur in many applications. Particularly, probability functions present a natural form to deal with this situation. Nevertheless, in some cases, the resulting probability functions are nonsmooth. This motivates us to propose a regularization employing the Moreau envelope of a scalar representation of the vector inequality. More precisely, we consider a probability function which covers most of the general classes of probabilistic constraints: $$φ(x)=\mathbb{P}(Φ(x,ξ)\in -\mathcal{K}),$$ where $\mathcal{K}$ is a convex cone of a Banach s…
Given a square matrix $A$ and a polynomial $p$, the Crouzeix ratio is the norm of the polynomial on the field of values of $A$ divided by the 2-norm of the matrix $p(A)$. Crouzeix's conjecture states that the globally minimal value of the Crouzeix ratio is 0.5, regardless of the matrix order and polynomial degree, and it is known that 1 is a frequently occurring locally minimal value. Making use of a heavy-tailed distribution to initialize our optimization computations, we demonstrate for the first time that the Crouzeix ratio has many other locally minimal values between 0.5 and 1. Besides sh…
This paper is devoted to the study of tilt stability of local minimizers for classical nonlinear programs with equality and inequality constraints in finite dimensions described by twice continuously differentiable functions. The importance of tilt stability has been well recognized from both theoretical and numerical perspectives of optimization, and this area of research has drawn much attention in the literature, especially in recent years. Based on advanced techniques of variational analysis and generalized differentiation, we derive here complete pointbased second-order characterizations …
This paper investigates the tilt stability of local minimizers for nonlinear programs under the relaxed constant rank constraint qualification in finite dimensions. By employing a neighborhood primal-dual approach and extending calculus rules for subgradient graphical derivative, we obtain some pointbased characterizations of tilt-stable local minimizers along with an explicit formula for calculating the exact bound of tilt stability. These results extend the corresponding ones of H. Gfrerer and B.S.Mordukhovich [SIAM J. Optim. 25 (2015), 2081-2119] by relaxing the constraint qualification and…
This paper is devoted to the study of tilt stability in finite dimensional optimization via the approach of using the subgradient graphical derivative. We establish a new characterization of tilt-stable local minimizers for a broad class of unconstrained optimization problems in terms of a uniform positive definiteness of the subgradient graphical derivative of the objective function around the point in question. By applying this result to nonlinear programming under the metric subregularity constraint qualification, we derive a second-order characterization and several new sufficient conditio…
This paper presents a nonsmooth proximal point technique for convex optimization in a special class of Hadamard manifold called homogeneous domains of positivity. The method is based on the particularization of the Rham decomposition theorem to symmetric spaces and it is applicable to the computing of minimizers for convex functions. Homogeneous domains of positivity to be considered in this work are those of specially reducible type in a sense which is introduced and largely discussed along the paper. General aspects on the technique are shown as the convergence analysis to the inner iteratio…
This paper focus on the minimization of a possibly nonsmooth objective function over the Stiefel manifold. The existing approaches either lack efficiency or can only tackle prox-friendly objective functions. We propose a constraint dissolving function named NCDF and show that it has the same first-order stationary points and local minimizers as the original problem in a neighborhood of the Stiefel manifold. Furthermore, we show that the Clarke subdifferential of NCDF is easy to achieve from the Clarke subdifferential of the objective function. Therefore, various existing approaches for unconst…
We investigate a class of constrained sparse regression problem with cardinality penalty, where the feasible set is defined by box constraint, and the loss function is convex, but not necessarily smooth. First, we put forward a smoothing fast iterative hard thresholding (SFIHT) algorithm for solving such optimization problems, which combines smoothing approximations, extrapolation techniques and iterative hard thresholding methods. The extrapolation coefficients can be chosen to satisfy $\sup_k β_k=1$ in the proposed algorithm. We discuss the convergence behavior of the algorithm with differen…
We propose an accelerated forward-backward method with fast convergence rate for finding a minimizer of a decomposable nonsmooth convex function over a closed convex set, and name it smoothing accelerated proximal gradient (SAPG) algorithm. The proposed algorithm combines the smoothing method with the proximal gradient algorithm with extrapolation $\frac{k-1}{k+α-1}$ and $α>3$. The updating rule of smoothing parameter $μ_k$ is a smart scheme and guarantees the global convergence rate of $o(\ln^σk/k)$ with $σ\in(\frac{1}{2},1]$ on the objective function values. Moreover, we prove that the seque…
This paper sheds new light on several interrelated topics of second-order variational analysis, both in finite and infinite-dimensional settings. We establish new relationships between second-order growth conditions on functions, the basic properties of metric regularity and subregularity of the limiting subdifferential, tilt-stability of local minimizers, and positive-definiteness/semidefiniteness properties of the second-order subdifferential (or generalized Hessian).
In this paper, we show that higher-order optimality conditions can be obtain for arbitrary nonsmooth function. We introduce a new higher-order directional derivative and higher-order subdifferential of Hadamard type of a given proper extended real function. This derivative is consistent with the classical higher-order Fréchet directional derivative in the sense that both derivatives of the same order coincide if the last one exists. We obtain necessary and sufficient conditions of order $n$ ($n$ is a positive integer) for a local minimum and isolated local minimum of order $n$ in terms of thes…
We show that the subgradient method converges only to local minimizers when applied to generic Lipschitz continuous and subdifferentially regular functions that are definable in an o-minimal structure. At a high level, the argument we present is appealingly transparent: we interpret the nonsmooth dynamics as an approximate Riemannian gradient method on a certain distinguished submanifold that captures the nonsmooth activity of the function. In the process, we develop new regularity conditions in nonsmooth analysis that parallel the stratification conditions of Whitney, Kuo, and Verdier and ext…
Variable Bregman Majorization-Minimization algorithms for nonconvex nonsmooth optimization, with application to Poisson imaging
Université Paris-Saclay, CentraleSupélec, Inria, Centre pour la Vision Numérique（巴黎萨克雷大学、中央苏佩莱克学院、法国国家信息与自动化研究所、数字视觉中心）；Université Paris-Saclay, Laboratoire BioMaps, CEA（巴黎萨克雷大学、BioMaps实验室、法国原子能和替代能源委员会）
Proximal methods are known to identify the underlying substructure of nonsmooth optimization problems. Even more, in many interesting situations, the output of a proximity operator comes with its structure at no additional cost, and convergence is improved once it matches the structure of a minimizer. However, it is impossible in general to know whether the current structure is final or not; such highly valuable information has to be exploited adaptively. To do so, we place ourselves in the case where a proximal gradient method can identify manifolds of differentiability of the nonsmooth objec…
Higher order necessary conditions for a minimizer of an optimal control problem are generally obtained for systems whose dynamics is continuously differentiable in the state variable. Here, by making use of the notion of set-valued Lie bracket we obtain a Goh-type condition for a control affine system with Lipschitz continuous dynamics and unbounded controls. In order to manage the simultaneous lack of smoothness of the adjoint equation and of the Lie bracket-like variations we make use of the notion of Quasi Differential Quotient. We conclude the paper with a worked out example where the esta…
In a Hilbert setting we study the convergence properties of a second order in time dynamical system combining viscous and Hessian-driven damping with time scaling in relation with the minimization of a nonsmooth and convex function. The system is formulated in terms of the gradient of the Moreau envelope of the objective function with time-dependent parameter. We show fast convergence rates for the Moreau envelope and its gradient along the trajectory, and also for the velocity of the system. From here we derive fast convergence rates for the objective function along a path which is the image …
This paper considers a stochastic optimization problem over the fixed point sets of quasinonexpansive mappings on Riemannian manifolds. The problem enables us to consider Riemannian hierarchical optimization problems over complicated sets, such as the intersection of many closed convex sets, the set of all minimizers of a nonsmooth convex function, and the intersection of sublevel sets of nonsmooth convex functions. We focus on adaptive learning rate optimization algorithms, which adapt step-sizes (referred to as learning rates in the machine learning field) to find optimal solutions quickly. …
This paper tackles the unconstrained minimization of a class of nonsmooth and nonconvex functions that can be written as finite max-functions. A gradient and function-based sampling method is proposed which, under special circumstances, either moves superlinearly to a minimizer of the problem of interest or superlinearly improves the optimality certificate. Global and local convergence analysis are presented, as well as illustrative examples that corroborate and elucidate the obtained theoretical results.
The paper concerns the second-order generalized differentiation theory of variational analysis and new applications of this theory to some problems of constrained optimization in finitedimensional spaces. The main attention is paid to the so-called (full and partial) second-order subdifferentials of extended-real-valued functions, which are dual-type constructions generated by coderivatives of frst-order subdifferential mappings. We develop an extended second-order subdifferential calculus and analyze the basic second-order qualification condition ensuring the fulfillment of the principal seco…
Modern statistical applications often involve minimizing an objective function that may be nonsmooth and/or nonconvex. This paper focuses on a broad Bregman-surrogate algorithm framework including the local linear approximation, mirror descent, iterative thresholding, DC programming and many others as particular instances. The recharacterization via generalized Bregman functions enables us to construct suitable error measures and establish global convergence rates for nonconvex and nonsmooth objectives in possibly high dimensions. For sparse learning problems with a composite objective, under …
We show that, for a fixed order $γ\geq 1$, each local minimizer of a rather general nonsmooth optimization problem in Euclidean spaces is either M-stationary in the classical sense (corresponding to stationarity of order $1$), satisfies stationarity conditions in terms of a coderivative construction of order $γ$, or is approximately stationary with respect to a critical direction as well as $γ$ in a certain sense. By ruling out the latter case with a constraint qualification not stronger than directional metric subregularity, we end up with new necessary optimality conditions comprising a mixt…
Note that the well-posedness of a proper lower semicontinuous function $f$ can be equivalently described using an admissible function. In the case when the objective function $f$ undergos the tilt perturbations in the sense of Poliquin and Rockafellar, adopting admissible functions $φ$ and $ψ$, this paper introduces and studies the stable well-posedness of $f$ with respect to $φ$ (in breif, $φ$-SLWP) and tilt-stable local minimum of $f$ with respect to $ψ$ (in brief, $ψ$-TSLM). In the special case when $φ(t)=t^2$ and $ψ(t)=t$, the corresponding $φ$-SLWP and $ψ$-TSLM reduce to the stable second…
Tsinghua University（清华大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Many optimization algorithms converge to stationary points. When the underlying problem is nonconvex, they may get trapped at local minimizers and occasionally stagnate near saddle points. We propose the Run-and-Inspect Method, which adds an "inspect" phase to existing algorithms that helps escape from non-global stationary points. The inspection samples a set of points in a radius $R$ around the current point. When a sample point yields a sufficient decrease in the objective, we move there and resume an existing algorithm. If no sufficient decrease is found, the current point is called an app…
Proximal splitting algorithms are well suited to solving large-scale nonsmooth optimization problems, in particular those arising in machine learning. We propose a new primal-dual algorithm, in which the dual update is randomized; equivalently, the proximity operator of one of the function in the problem is replaced by a stochastic oracle. For instance, some randomly chosen dual variables, instead of all, are updated at each iteration. Or, the proximity operator of a function is called with some small probability only. A nonsmooth variance-reduction technique is implemented so that the algorit…
The paper is devoted to the study, characterizations, and applications of variational convexity of functions, the property that has been recently introduced by Rockafellar together with its strong counterpart. First we show that these variational properties of an extended-real-valued function are equivalent to, respectively, the conventional (local) convexity and strong convexity of its Moreau envelope. Then we derive new characterizations of both variational convexity and variational strong convexity of general functions via their second-order subdifferentials (generalized Hessians), which ar…
