In this paper we study the right differentiability of a parametric infimum function over a parametric set defined by equality constraints. We present a new theorem with sufficient conditions for the right differentiability with respect to the parameter. Target applications are nonconvex objective functions with equality constraints arising in optimal control and shape optimisation. The theorem makes use of the averaged adjoint approach in conjunction with the variational approach of Kunisch, Ito and Peichl. We provide two examples of our abstract result: (a) a shape optimisation problem involving a semilinear partial differential equation which exhibits infinitely many solutions, (b) a finite dimensional quadratic function subject to a nonlinear equation.
In this article we propose a descent method for equality and inequality constrained multiobjective optimization problems (MOPs) which generalizes the steepest descent method for unconstrained MOPs by Fliege and Svaiter to constrained problems by using two active set strategies. Under some regularity assumptions on the problem, we show that accumulation points of our descent method satisfy a necessary condition for local Pareto optimality. Finally, we show the typical behavior of our method in a numerical example.
In this paper, we propose an inertial accelerated primal-dual method for the linear equality constrained convex optimization problem. When the objective function has a ``nonsmooth + smooth'' composite structure, we further propose an inexact inertial primal-dual method by linearizing the smooth individual function and solving the subproblem inexactly. Assuming merely convexity, we prove that the proposed methods enjoy $\mathcal{O}(1/k^2)$ convergence rate on the objective residual and the feasibility violation in the primal model. Numerical results are reported to demonstrate the validity of t…
In this paper, we propose a stochastic method for solving equality constrained optimization problems that utilizes predictive variance reduction. Specifically, we develop a method based on the sequential quadratic programming paradigm that employs variance reduction in the gradient approximations. Under reasonable assumptions, we prove that a measure of first-order stationarity evaluated at the iterates generated by our proposed algorithm converges to zero in expectation from arbitrary starting points, for both constant and adaptive step size strategies. Finally, we demonstrate the practical p…
In this paper, a proof of asymptotic stability for the combined system-optimizer dynamics associated with a class of real-time methods for equality constrained nonlinear model predictive control is presented. General Q-linearly convergent online optimization methods are considered and asymptotic stability results are derived for the case where a single iteration of the optimizer is carried out per sampling time. In particular, it is shown that, if the underlying sampling time is sufficiently short, asymptotic stability can be guaranteed. The results constitute an extension to existing attracti…
We extend the class of SQP methods for equality constrained optimization to the setting of differentiable manifolds. The use of retractions and stratifications allows us to pull back the involved mappings to linear spaces. We study local quadratic convergence to minimizers. In addition we present a composite step method for globalization based on cubic regularization of the objective function and affine covariant damped Newton method for feasibility. We show transition to fast local convergence of this scheme. We test our method on equilibrium problems in finite elasticity where the stable equ…
We provide a generalization of first-order necessary conditions of optimality for infinite-dimensional optimization problems with a finite number of inequality constraints and with a finite number of inequality and equality constraints. Our assumptions on the differentiability of the functions are weaker than those of existing results.
Second-order dynamical systems are important tools for solving optimization problems, and most of existing works in this field have focused on unconstrained optimization problems. In this paper, we propose an inertial primal-dual dynamical system with constant viscous damping and time scaling for the linear equality constrained convex optimization problem, which consists of a second-order ODE for the primal variable and a first-order ODE for the dual variable. When the scaling satisfies certain conditions, we prove its convergence property without assuming strong convexity. Even the convergenc…
A worst-case complexity bound is proved for a sequential quadratic optimization (commonly known as SQP) algorithm that has been designed for solving optimization problems involving a stochastic objective function and deterministic nonlinear equality constraints. Barring additional terms that arise due to the adaptivity of the monotonically nonincreasing merit parameter sequence, the proved complexity bound is comparable to that known for the stochastic gradient algorithm for unconstrained nonconvex optimization. The overall complexity bound, which accounts for the adaptivity of the merit param…
We establish new results of first-order necessary conditions of optimality for finite-dimensional problems with inequality constraints and for problems with equality and inequality constraints, in the form of John's theorem and in the form of Karush-Kuhn-Tucker's theorem. In comparison with existing results we weaken assumptions of continuity and of differentiability.
Tikhonov regularized second-order plus first-order primal-dual dynamical systems with asymptotically vanishing damping for linear equality constrained convex optimization problems
A class of convex constrained minimization problems over polyhedral cones for geometry-dependent quadratic objective functions is considered in a functional analysis framework. Shape differentiability of the primal minimization problem needs a bijective property for mapping of the primal cone. This restrictive assumption is relaxed to bijection of the dual cone within the Lagrangian formulation as a primal-dual minimax problem. In this paper, we give results on primal-dual shape sensitivity analysis that extends the class of shape-differentiable problems supported by explicit formula of the sh…
This paper presents a methodology for using varying sample sizes in sequential quadratic programming (SQP) methods for solving equality constrained stochastic optimization problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the gradient in conjunction with inexact solutions to the SQP subproblems. Under reasonable assumptions on the quality of the employed gradient approximations and the accuracy of the solutions to the SQP subproblems, we establish global convergence results for the proposed method. Motivated by these results, t…
We prove convergence of a single time-scale stochastic subgradient method with subgradient averaging for constrained problems with a nonsmooth and nonconvex objective function having the property of generalized differentiability. As a tool of our analysis, we also prove a chain rule on a path for such functions.
In this paper we consider the minimization of a continuous function that is potentially not differentiable or not twice differentiable on the boundary of the feasible region. By exploiting an interior point technique, we present first- and second-order optimality conditions for this problem that reduces to classical ones when the derivative on the boundary is available. For this type of problems, existing necessary conditions often rely on the notion of subdifferential or become non-trivially weaker than the KKT condition in the (twice-)differentiable counterpart problems. In contrast, this pa…
The classic Alternating Direction Method of Multipliers (ADMM) is a popular framework to solve linear-equality constrained problems. In this paper, we extend the ADMM naturally to nonlinear equality-constrained problems, called neADMM. The difficulty of neADMM is to solve nonconvex subproblems. We provide globally optimal solutions to them in two important applications. Experiments on synthetic and real-world datasets demonstrate excellent performance and scalability of our proposed neADMM over existing state-of-the-start methods.
Verifying the Second-Order Sufficient Condition (SOSC), thus ensuring a stationary point locally minimizes a given objective function (subject to certain constraints), is an essential component of non-convex computational optimization and equilibrium programming. This article proposes three new "Hessian-free" tests of the SOSC that can be implemented efficiently with gradient evaluations alone and reveal feasible directions of negative curvature when the SOSC fails. The Bordered Hessian Test and a Matrix Inertia test, two classical tests of the SOSC, require explicit knowledge of the Hessian o…
Kalman Filtering problems often have inherent and known constraints in the physical dynamics that are not exploited despite potentially significant gains (e.g., fixed speed of a motor). In this paper, we review existing methods and propose some new ideas for filtering in the presence of equality constraints. We then show that three methods for incorporating state space equality constraints are mathematically equivalent to the more general "Projection" method, which allows different weighting matrices when projecting the estimate. Still, the different approaches have advantages in implementatio…
This paper discusses a special kind of convex constrained optimization problem, whose constraints consist of box inequalities and linear equalities. For this problem, in addition to general optimization algorithms such as exact penalty algorithm and interior point algorithm, there is a simple iterative algorithm that is simple to implement, which is favored by machine learning practitioners.
Generalized Differentiability and Second-Order Necessary Optimality Conditions for an Elliptic Optimal Control Problem with Exponential Nonlinearity and Discrete Measures
In this work, we deal with rank-constrained integer least-squares optimization problems arising in low-rank matrix factorization related applications. We propose a solution for constrained integer least-squares problem subject to equality, sparsity, and rank constraints. The algorithm combines the Fincke-Pohst enumeration (or sphere decoding algorithm) with rank constraints and sparse solutions of Diophantine equations to arrive at an optimal solution. The proposed approach consists of two steps as follows: (i) find the solution set for Diophantine equations arising from the linear and sparsit…
In this paper, we are concerned with optimal control problems evolved on Riemannian manifolds, where the initial and final states satisfy some inequality and equality type constraints, and the control set is a separable metric space. We obtain the second order necessary conditions of integral and quasi-pointwise forms, both of which work for Pontryagin type critical controls and involve the curvature tensor. Also, we apply the condition of integral form to the Bolza problem, where the initial and final states are subject to equality's type constraint.
A sequential quadratic optimization algorithm is proposed for solving smooth nonlinear equality constrained optimization problems in which the objective function is defined by an expectation of a stochastic function. The algorithmic structure of the proposed method is based on a step decomposition strategy that is known in the literature to be widely effective in practice, wherein each search direction is computed as the sum of a normal step (toward linearized feasibility) and a tangential step (toward objective decrease in the null space of the constraint Jacobian). However, the proposed meth…
First-order methods have been studied for nonlinear constrained optimization within the framework of the augmented Lagrangian method (ALM) or penalty method. We propose an improved inexact ALM (iALM) and conduct a unified analysis for nonconvex problems with either affine equality or nonconvex constraints. Under certain regularity conditions (that are also assumed by existing works), we show an $\tilde{O}(\varepsilon^{-\frac{5}{2}})$ complexity result for a problem with a nonconvex objective and affine equality constraints and an $\tilde{O}(\varepsilon^{-3})$ complexity result for a problem wi…
This paper presents a state and state-input constrained variant of the discrete-time iterative Linear Quadratic Regulator (iLQR) algorithm, with linear time-complexity in the number of time steps. The approach is based on a projection of the control input onto the nullspace of the linearized constraints. We derive a fully constraint-compliant feedforward-feedback control update rule, for which we can solve efficiently with Riccati-style difference equations. We assume that the relative degree of all constraints in the discrete-time system model is equal to one, which often holds for robotics p…
We investigate optimality conditions for optimization problems constrained by a class of variational inequalities of the second kind. Based on a nonsmooth primal-dual reformulation of the governing inequality, the differentiability of the solution map is studied. Directional differentiability is proved both for finite-dimensional and function space problems, under suitable assumptions on the active set. A characterization of B- and strong stationary optimal solutions is obtained thereafter. Finally, based on the obtained first-order information, a trust-region algorithm is proposed for the sol…
In this work we propose a framework to address the issue of state dependent nonlinear equality-constrained state estimation using Bayesian filtering. This framework is constructed specifically for a linear approximation of Bayesian filtering that uses the theory of Optimal Transport. As a part of this framework, we present three traditionally-used nonlinear equality constraint-preserving algorithms coupled with the Optimal Transport based filter: the equality-constrained Optimal Transport filter, the projected Optimal Transport filter, and the measurement-augmented Optimal Transport filter. In…
In unconstrained maximum a posteriori (MAP) and maximum likelihood estimation, the inverse of minus the merit-function Hessian matrix is an approximation of the estimate covariance matrix. In the Bayesian context of MAP estimation, it is the covariance of a normal approximation of the posterior around the mode; while in maximum likelihood estimation, it an approximation of the inverse Fisher information matrix, to which the covariance of efficient estimators converge. These measures are routinely used in system identification to evaluate the estimate uncertainties and diagnose problems such as…
Stochastic bilevel optimization, which captures the inherent nested structure of machine learning problems, is gaining popularity in many recent applications. Existing works on bilevel optimization mostly consider either unconstrained problems or constrained upper-level problems. This paper considers the stochastic bilevel optimization problems with equality constraints both in the upper and lower levels. By leveraging the special structure of the equality constraints problem, the paper first presents an alternating implicit projected SGD approach and establishes the $\tilde{\cal O}(ε^{-2})$ s…
We develop a general equality-constrained nonlinear optimization algorithm based on a smooth penalty function proposed by Fletcher (1970). Although it was historically considered to be computationally prohibitive in practice, we demonstrate that the computational kernels required are no more expensive than other widely accepted methods for nonlinear optimization. The main kernel required to evaluate the penalty function and its derivatives is solving a structured linear system. We show how to solve this system efficiently by storing a single factorization each iteration when the matrices are a…
A method is proposed for solving equality constrained nonlinear optimization problems involving twice continuously differentiable functions. The method employs a trust funnel approach consisting of two phases: a first phase to locate an $ε$-feasible point and a second phase to seek optimality while maintaining at least $ε$-feasibility. A two-phase approach of this kind based on a cubic regularization methodology was recently proposed along with a supporting worst-case iteration complexity analysis. Unfortunately, however, in that approach, the objective function is completely ignored in the fi…
University of Minnesota（明尼苏达大学）；The Hong Kong Polytechnic University（香港理工大学）
In this paper we consider finding a second-order stationary point (SOSP) of nonconvex equality constrained optimization when a nearly feasible point is known. In particular, we first propose a new Newton-CG method for finding an approximate SOSP of unconstrained optimization and show that it enjoys a substantially better complexity than the Newton-CG method [56]. We then propose a Newton-CG based augmented Lagrangian (AL) method for finding an approximate SOSP of nonconvex equality constrained optimization, in which the proposed Newton-CG method is used as a subproblem solver. We show that und…
University of Michigan（密歇根大学）；Lehigh University（里海大学）
Sequential quadratic optimization algorithms are proposed for solving smooth nonlinear optimization problems with equality constraints. The main focus is an algorithm proposed for the case when the constraint functions are deterministic, and constraint function and derivative values can be computed explicitly, but the objective function is stochastic. It is assumed in this setting that it is intractable to compute objective function and derivative values explicitly, although one can compute stochastic function and gradient estimates. As a starting point for this stochastic setting, an algorith…
We consider the problem of minimizing a non-convex function over a smooth manifold $\mathcal{M}$. We propose a novel algorithm, the Orthogonal Directions Constrained Gradient Method (ODCGM) which only requires computing a projection onto a vector space. ODCGM is infeasible but the iterates are constantly pulled towards the manifold, ensuring the convergence of ODCGM towards $\mathcal{M}$. ODCGM is much simpler to implement than the classical methods which require the computation of a retraction. Moreover, we show that ODCGM exhibits the near-optimal oracle complexities $\mathcal{O}(1/\varepsil…
