A worst-case complexity bound is proved for a sequential quadratic optimization (commonly known as SQP) algorithm that has been designed for solving optimization problems involving a stochastic objective function and deterministic nonlinear equality constraints. Barring additional terms that arise due to the adaptivity of the monotonically nonincreasing merit parameter sequence, the proved complexity bound is comparable to that known for the stochastic gradient algorithm for unconstrained nonconvex optimization. The overall complexity bound, which accounts for the adaptivity of the merit parameter sequence, shows that a result comparable to the unconstrained setting (with additional logarithmic factors) holds with high probability.
We extend the class of SQP methods for equality constrained optimization to the setting of differentiable manifolds. The use of retractions and stratifications allows us to pull back the involved mappings to linear spaces. We study local quadratic convergence to minimizers. In addition we present a composite step method for globalization based on cubic regularization of the objective function and affine covariant damped Newton method for feasibility. We show transition to fast local convergence of this scheme. We test our method on equilibrium problems in finite elasticity where the stable equ…
This paper presents a methodology for using varying sample sizes in sequential quadratic programming (SQP) methods for solving equality constrained stochastic optimization problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the gradient in conjunction with inexact solutions to the SQP subproblems. Under reasonable assumptions on the quality of the employed gradient approximations and the accuracy of the solutions to the SQP subproblems, we establish global convergence results for the proposed method. Motivated by these results, t…
In this work, based on the ideas of alternating direction method with multipliers (ADMM) and sequential quadratic programming (SQP), as well as Armijo line search technology, monotone splitting SQP algorithms for two-block nonconvex optimization problems with linear equality, inequality and box constraints are discussed. Firstly, the discussed problem is transformed into an optimization problem with only linear equality and box constraints by introducing slack variables. Secondly, we use the idea of ADMM to decompose the quadratic programming (QP) subproblem. Especially, the QP subproblem corr…
We prove that a "first-order" Sequential Quadratic Programming (SQP) algorithm for equality constrained optimization has local linear convergence with rate $(1-1/κ_R)^k$, where $κ_R$ is the condition number of the Riemannian Hessian, and global convergence with rate $k^{-1/4}$. Our analysis builds on insights from Riemannian optimization -- we show that the SQP and Riemannian gradient methods have nearly identical behavior near the constraint manifold, which could be of broader interest for understanding constrained optimization.
The problem of interest is the minimization of a nonlinear function subject to nonlinear equality constraints using a sequential quadratic programming (SQP) method. The minimization must be performed while observing only noisy evaluations of the objective and constraint functions. In order to obtain stability, the classical SQP method is modified by relaxing the standard Armijo line search based on the noise level in the functions, which is assumed to be known. Convergence theory is presented giving conditions under which the iterates converge to a neighborhood of the solution characterized by…
We consider solving nonlinear optimization problems with a stochastic objective and deterministic equality constraints. We assume for the objective that its evaluation, gradient, and Hessian are inaccessible, while one can compute their stochastic estimates by, for example, subsampling. We propose a stochastic algorithm based on sequential quadratic programming (SQP) that uses a differentiable exact augmented Lagrangian as the merit function. To motivate our algorithm design, we first revisit and simplify an old SQP method \citep{Lucidi1990Recursive} developed for solving deterministic problem…
Redundancy is related to the amount of functionality that the structure can sustain in the worst-case scenario of structural degradation. This paper proposes a widely-applicable concept of redundancy optimization of finite-dimensional structures. The concept is consistent with the robust structural optimization, as well as the quantitative measure of structural redundancy based on the information-gap theory. A derivative-free algorithm is proposed based on the sequential quadratic programming (SQP) method, where we use the finite-difference method with adaptively varying the difference increme…
This thesis studies derivative-free optimization (DFO), particularly model-based methods and software. These methods are motivated by optimization problems for which it is impossible or prohibitively expensive to access the first-order information of the objective function and possibly the constraint functions. In particular, this thesis presents PDFO, a package we develop to provide both MATLAB and Python interfaces to Powell's model-based DFO solvers, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA. Moreover, a significant part of this thesis is devoted to developing a new DFO method based…
Projected Gradient Descent denotes a class of iterative methods for solving optimization programs. Its applicability to convex optimization programs has gained significant popularity for its intuitive implementation that involves only simple algebraic operations. In fact, if the projection onto the feasible set is easy to compute, then the method has low complexity. On the other hand, when the problem is nonconvex, e.g. because of nonlinear equality constraints, the projection becomes hard and thus impractical. In this paper, we propose a projected gradient method for Nonlinear Programs (NLPs)…
In this paper, a class of general nonlinear programming problems with inequality and equality constraints is discussed. Firstly, the original problem is transformed into an associated simpler equivalent problem with only inequality constraints. Then, inspired by the ideals of sequential quadratic programming (SQP) method and the method of system of linear equations (SLE), a new type of SQP algorithm for solving the original problem is proposed. At each iteration, the search direction is generated by the combination of two directions, which are obtained by solving an always feasible quadratic p…
University of Michigan（密歇根大学）；Lehigh University（里海大学）
Sequential quadratic optimization algorithms are proposed for solving smooth nonlinear optimization problems with equality constraints. The main focus is an algorithm proposed for the case when the constraint functions are deterministic, and constraint function and derivative values can be computed explicitly, but the objective function is stochastic. It is assumed in this setting that it is intractable to compute objective function and derivative values explicitly, although one can compute stochastic function and gradient estimates. As a starting point for this stochastic setting, an algorith…
The nonlinear optimization problem with linear constraints has many applications in engineering fields such as the visual-inertial navigation and localization of an unmanned aerial vehicle maintaining the horizontal flight. In order to solve this practical problem efficiently, this paper constructs a continuation method with the trusty time-stepping scheme for the linearly equality-constrained optimization problem at every sampling time. At every iteration, the new method only solves a system of linear equations other than the traditional optimization method such as the sequential quadratic pr…
We analyze worst-case complexity of a Proximal augmented Lagrangian (Proximal AL) framework for nonconvex optimization with nonlinear equality constraints. When an approximate first-order (second-order) optimal point is obtained in the subproblem, an $ε$ first-order (second-order) optimal point for the original problem can be guaranteed within $\mathcal{O}(1/ ε^{2 - η})$ outer iterations (where $η$ is a user-defined parameter with $η\in[0,2]$ for the first-order result and $η\in [1,2]$ for the second-order result) when the proximal term coefficient $β$ and penalty parameter $ρ$ satisfy $β= \ma…
A method is proposed for solving equality constrained nonlinear optimization problems involving twice continuously differentiable functions. The method employs a trust funnel approach consisting of two phases: a first phase to locate an $ε$-feasible point and a second phase to seek optimality while maintaining at least $ε$-feasibility. A two-phase approach of this kind based on a cubic regularization methodology was recently proposed along with a supporting worst-case iteration complexity analysis. Unfortunately, however, in that approach, the objective function is completely ignored in the fi…
This paper considers the regularization continuation method and the trust-region updating strategy for the optimization problem with linear equality constraints.The proposed method utilizes the linear conservation law of the regularization continuation method such that it does not need to compute the correction step for preserving the feasibility other than the previous continuation methods and the quasi-Newton updating formulas for the linearly constrained optimization problem. Moreover, the new method uses the special limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) formula as the pr…
The augmented Lagrangian method (ALM) is a benchmark for convex programming problems with linear constraints; ALM and its variants for linearly equality-constrained convex minimization models have been well studied in the literature. However, much less attention has been paid to ALM for efficiently solving linearly inequality-constrained convex minimization models. In this paper, we exploit an enlightening reformulation of the newly developed indefinite linearized ALM for the equality-constrained convex optimization problem, and present a new indefinite linearized ALM scheme for efficiently so…
Dynamic-Threshold Algorithms for the Continuous Quadratic Knapsack Problem: Reset Mechanisms and Complexity
Center for Applied Mathematics of Fujian Province, School of Mathematics and Statistics, Fuzhou University（福建省应用数学中心，福州大学数学与统计学院）；School of Mathematics and Statistics, Fuzhou University（福州大学数学与统计学院）
In this paper we study the worst-case complexity of an inexact Augmented Lagrangian method for nonconvex constrained problems. Assuming that the penalty parameters are bounded, we prove a complexity bound of $\mathcal{O}(|\log(ε)|)$ outer iterations for the referred algorithm to generate an $ε$-approximate KKT point, for $ε\in (0,1)$. When the penalty parameters are unbounded, we prove an outer iteration complexity bound of $\mathcal{O}\left(ε^{-2/(α-1)}\right)$, where $α>1$ controls the rate of increase of the penalty parameters. For linearly constrained problems, these bounds yield to evalua…
