Nonconvex optimization problems with an L1-constraint are ubiquitous, and are found in many application domains including: optimal control of hybrid systems, machine learning and statistics, and operations research. This paper shows that nonconvex optimization problems with an L1-constraint can be approximately solved in polynomial time. We first show that nonlinear integer programs with an L1-constraint can be solved in a number of oracle steps that is polynomial in the dimension of the decision variable, for each fixed radius of the L1-constraint. When specialized to polynomial integer programs, our result shows that such problems have a time complexity that is polynomial in simultaneously both the dimension of the decision variables and number of constraints, for each fixed radius of the L1-constraint. We prove this result using a geometric argument that leverages ideas from stochastic process theory and from the theory of convex bodies in high-dimensional spaces. We conclude by providing an additive polynomial time approximation scheme (PTAS) for continuous optimization of Lipschitz functions subject to Lipschitz constraints intersected with an L1-constraint, and we sketch a generalization to mixed-integer optimization.
This paper presents a canonical dual approach for solving a nonconvex global optimization problem governed by a sum of fourth-order polynomial and a log-sum-exp function. Such a problem arises extensively in engineering and sciences. Based on the canonical duality-triality theory, this nonconvex problem is transformed to an equivalent dual problem, which can be solved easily under certain conditions. We proved that both global minimizer and the biggest local extrema of the primal problem can be obtained analytically from the canonical dual solutions. As two special cases, a quartic polynomial …
We present a geometrical analysis on the completely positive programming reformulation of quadratic optimization problems and its extension to polynomial optimization problems with a class of geometrically defined nonconvex conic programs and their covexification. The class of nonconvex conic programs is described with a linear objective functionin a linear space $V$, and the constraint set is represented geometrically as the intersection of a nonconvex cone $K \subset V$, a face $J$ of the convex hull of $K$ and a parallel translation $L$ of a supporting hyperplane of the nonconvex cone $K$. …
We consider the problem of approximating nonconvex quadratic optimization with ellipsoid constraints (ECQP). We show some SDP-based approximation bounds for special cases of (ECQP) can be improved by trivially applying the extened Pataki's procedure. The main result of this paper is to give a new analysis on approximating (ECQP) by the SDP relaxation, which greatly improves Tseng's result [SIAM Journal Optimization, 14, 268-283, 2003]. As an application, we strictly improve the approximation ratio for the assignment-polytope constrained quadratic program.
A second-order cone representable class of nonconvex quadratic programs
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院 H. 米尔顿·斯图尔特工业与系统工程学）；Department of Industrial and Systems Engineering, Lehigh University（利哈伊大学工业与系统工程学系）
We consider the problem of decomposing a multivariate polynomial as the difference of two convex polynomials. We introduce algebraic techniques which reduce this task to linear, second order cone, and semidefinite programming. This allows us to optimize over subsets of valid difference of convex decompositions (dcds) and find ones that speed up the convex-concave procedure (CCP). We prove, however, that optimizing over the entire set of dcds is NP-hard.
Yuan's theorem of the alternative is an important theoretical tool in optimization, which provides a checkable certificate for the infeasibility of a strict inequality system involving two homogeneous quadratic functions. In this paper, we provide a tractable extension of Yuan's theorem of the alternative to the symmetric tensor setting. As an application, we establish that the optimal value of a class of nonconvex polynomial optimization problems with suitable sign structure (or more explicitly, with essentially non-positive coefficients) can be computed by a related convex conic programming …
Decades of advances in mixed-integer linear programming (MILP) and recent development in mixed-integer second-order-cone programming (MISOCP) have translated very mildly to progresses in global solving nonconvex mixed-integer quadratically constrained programs (MIQCP). In this paper we propose a new approach, namely Compact Disjunctive Approximation (CDA), to approximate nonconvex MIQCP to arbitrary precision by \textit{convex} MIQCPs, which can be solved by MISOCP solvers. For nonconvex MIQCP with $n$ variables and $m$ general quadratic constraints, our method yields relaxations with at most …
We study a canonical duality method to solve a mixed-integer nonconvex fourth-order polynomial minimization problem with fixed cost terms. This constrained nonconvex problem can be transformed into a continuous concave maximization dual problem without duality gap. The global optimality conditions are proposed and the existence and uniqueness criteria are discussed. Application to a decoupled mixed-integer problem is illustrated and analytic solution for a global minimum is obtained under some suitable conditions. Several examples are given to show the method is effective.
This paper considers a class of constrained stochastic composite optimization problems whose objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a certain non-differentiable (but convex) component. In order to solve these problems, we propose a randomized stochastic projected gradient (RSPG) algorithm, in which proper mini-batch of samples are taken at each iteration depending on the total budget of stochastic samples allowed. The RSPG algorithm also employs a general distance function to allow taking advantage of the geometry of the f…
This paper presents smoothing schemes for obtaining approximate stationary points of unconstrained or linearly-constrained composite nonconvex-concave min-max (and hence nonsmooth) problems by applying well-known algorithms to composite smooth approximations of the original problems. More specifically, in the unconstrained (resp. constrained) case, approximate stationary points of the original problem are obtained by applying, to its composite smooth approximation, an accelerated inexact proximal point (resp. quadratic penalty) method presented in a previous paper by the authors. Iteration com…
The $\ell_1$ norm is the tight convex relaxation for the $\ell_0$ "norm" and has been successfully applied for recovering sparse signals. For problems with fewer samplings, one needs to enhance the sparsity by nonconvex penalties such as $\ell_p$ "norm". As one method for solving $\ell_p$ minimization problems, iteratively reweighted $\ell_1$ minimization updates the weight for each component based on the value of the same component at the previous iteration. It assigns large weights on small components in magnitude and small weights on large components in magnitude. In this paper, we consider…
In this paper, we consider the constraint set $K$ of inequalities with nonsmooth nonconvex constraint functions. We show that under Abadie's constraint qualification the "perturbation property" of the best approximation to any $x$ in $\R^n$ from a convex set $\tK:=C \cap K$ is characterized by the strong conical hull intersection property (strong CHIP) of $C$ and $K,$ where $C$ is a non-empty closed convex subset of $\R^n$ and the set $K$ is represented by $K:=\{x\in \R^n : g_j(x) \le 0, \ \forall \ j=1,2,\ldots,m \}$ with $g_j : \R^n \lrar \R$ $(j=1,2, \cdots,m)$ is a tangentially convex func…
Algorithms for solving nonconvex, nonsmooth, finite-sum optimization problems are proposed and tested. In particular, the algorithms are proposed and tested in the context of an optimization problem formulation arising in semi-supervised machine learning. The common feature of all algorithms is that they employ an incremental quasi-Newton (IQN) strategy, specifically an incremental BFGS (IBFGS) strategy. One applies an IBFGS strategy to the problem directly, whereas the others apply an IBFGS strategy to a difference-of-convex reformulation, smoothed approximation, or (strongly) convex local ap…
In this work, we propose a new local optimization method to solve a class of nonconvex semidefinite programming (SDP) problems. The basic idea is to approximate the feasible set of the nonconvex SDP problem by inner positive semidefinite convex approximations via a parameterization technique. This leads to an iterative procedure to search a local optimum of the nonconvex problem. The convergence of the algorithm is analyzed under mild assumptions. Applications in static output feedback control are benchmarked and numerical tests are implemented based on the data from the COMPLeib library.
We study a class of quadratically constrained quadratic programs (QCQPs), called {\em diagonal QCQPs\/}, which contain no off-diagonal terms $x_j x_k$ for $j \ne k$, and we provide a sufficient condition on the problem data guaranteeing that the basic Shor semidefinite relaxation is exact. Our condition complements and refines those already present in the literature and can be checked in polynomial time. We then extend our analysis from diagonal QCQPs to general QCQPs, i.e., ones with no particular structure. By reformulating a general QCQP into diagonal form, we establish new, polynomial-time…
We estimate $n$ phases (angles) from noisy pairwise relative phase measurements. The task is modeled as a nonconvex least-squares optimization problem. It was recently shown that this problem can be solved in polynomial time via convex relaxation, under some conditions on the noise. In this paper, under similar but more restrictive conditions, we show that a modified version of the power method converges to the global optimum. This is simpler and (empirically) faster than convex approaches. Empirically, they both succeed in the same regime. Further analysis shows that, in the same noise regime…
Consider the problem of minimizing the sum of a smooth (possibly non-convex) and a convex (possibly nonsmooth) function involving a large number of variables. A popular approach to solve this problem is the block coordinate descent (BCD) method whereby at each iteration only one variable block is updated while the remaining variables are held fixed. With the recent advances in the developments of the multi-core parallel processing technology, it is desirable to parallelize the BCD method by allowing multiple blocks to be updated simultaneously at each iteration of the algorithm. In this work, …
Chance-constrained programs (CCPs) constitute a difficult class of stochastic programs due to its possible nondifferentiability and nonconvexity even with simple linear random functionals. Existing approaches for solving the CCPs mainly deal with convex random functionals within the probability function. In the present paper, we consider two generalizations of the class of chance constraints commonly studied in the literature; one generalization involves probabilities of disjunctive nonconvex functional events and the other generalization involves mixed-signed affine combinations of the result…
We study distributed stochastic nonconvex optimization in multi-agent networks. We introduce a novel algorithmic framework for the distributed minimization of the sum of the expected value of a smooth (possibly nonconvex) function (the agents' sum-utility) plus a convex (possibly nonsmooth) regularizer. The proposed method hinges on successive convex approximation (SCA) techniques, leveraging dynamic consensus as a mechanism to track the average gradient among the agents, and recursive averaging to recover the expected gradient of the sum-utility function. Almost sure convergence to (stationar…
School of Industrial and Systems Engineering at Georgia Tech（佐治亚理工学院工业与系统工程学院）
Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points, but hurts…
The concept of dispatchable region is useful in quantifying how much renewable generation power the system can handle. In this paper, we aim to provide an improved dispatchable region approximation method in distribution networks. First, based on the nonlinear Dist-Flow model, an optimization problem that minimizes the sum of slack variables is formulated to describe the dispatchable region. The nonconvexity caused by alternating-current (AC) power flow constraints makes it intractable. To deal with this issue, the problem is relaxed to a second-order cone program (SOCP) whose strong dual prob…
Given two arbitrary closed sets in Euclidean space, a simple transversality condition guarantees that the method of alternating projections converges locally, at linear rate, to a point in the intersection. Exact projection onto nonconvex sets is typically intractable, but we show that computationally-cheap inexact projections may suffice instead. In particular, if one set is defined by sufficiently regular smooth constraints, then projecting onto the approximation obtained by linearizing those constraints around the current iterate suffices. On the other hand, if one set is a smooth manifold …
In this paper, we generalize the well-known Nesterov's accelerated gradient (AG) method, originally designed for convex smooth optimization, to solve nonconvex and possibly stochastic optimization problems. We demonstrate that by properly specifying the stepsize policy, the AG method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first-order information, similarly to the gradient descent method. We then consider an important class of composite optimization problems and show that the AG method can solve them uniformly, i.e., by us…
Consider the problem of minimizing the expected value of a (possibly nonconvex) cost function parameterized by a random (vector) variable, when the expectation cannot be computed accurately (e.g., because the statistics of the random variables are unknown and/or the computational complexity is prohibitive). Classical sample stochastic gradient methods for solving this problem may empirically suffer from slow convergence. In this paper, we propose for the first time a stochastic parallel Successive Convex Approximation-based (best-response) algorithmic framework for general nonconvex stochastic…
This work studies low-rank approximation of a positive semidefinite matrix from partial entries via nonconvex optimization. We characterized how well local-minimum based low-rank factorization approximates a fixed positive semidefinite matrix without any assumptions on the rank-matching, the condition number or eigenspace incoherence parameter. Furthermore, under certain assumptions on rank-matching and well-boundedness of condition numbers and eigenspace incoherence parameters, a corollary of our main theorem improves the state-of-the-art sampling rate results for nonconvex matrix completion …
The goal of this paper is to solve a class of high-order polynomial benchmark optimization problems, including the Goldstein-Price problem and the Three Hump Camel Back problem. By using a generalized canonical duality theory, we are able to transform the nonconvex primal problems to concave dual problems over convex domain(without duality gap), which can be solved easily to obtain global solutions.
Convex inner approximations of nonconvex semialgebraic sets applied to fixed-order controller design
CNRS（法国国家科学研究中心；实验室分析与系统研究所）；LAAS（图卢兹大学）；Université de Toulouse（捷克布拉格捷克理工大学电气工程学院）；UPS, INSA, INP, ISAE；UT1, UTM, LAAS；Faculty of Electrical Engineering, Czech Technical University in Prague
This work studies the problem of maximizing a higher degree real homogeneous multivariate polynomial over the unit sphere. This problem is equivalent to finding the leading eigenvalue of the associated symmetric tensor of higher order, which is nonconvex and NP-hard. Recent advances show that semidefinite relaxation is quite effective to find a global solution. However, the solution methods involve full/partial eigenvalue decomposition during the iterates, which heavily limits its efficiency and scalability. On the other hand, for odd degree (odd order) cases, the order has to be increased to …
We present complexity and numerical results for a new asynchronous parallel algorithmic method for the minimization of the sum of a smooth nonconvex function and a convex nonsmooth regularizer, subject to both convex and nonconvex constraints. The proposed method hinges on successive convex approximation techniques and a novel probabilistic model that captures key elements of modern computational architectures and asynchronous implementations in a more faithful way than state-of-the-art models. In the companion paper we provided a detailed description on the probabilistic model and gave conver…
We consider the minimization of a sum of an expectation-valued coordinate-wise $L_i$-smooth nonconvex function and a nonsmooth block-separable convex regularizer. We propose an asynchronous variance-reduced algorithm, where in each iteration, a single block is randomly chosen to update its estimates by a proximal variable sample-size stochastic gradient scheme, while the remaining blocks are kept invariant. Notably, each block employs a steplength that is in accordance with its block-specific Lipschitz constant while block-specific batch-sizes are random variables updated at a rate that grows …
This paper proposes and develops new linesearch methods with inexact gradient information for finding stationary points of nonconvex continuously differentiable functions on finite-dimensional spaces. Some abstract convergence results for a broad class of linesearch methods are stablished. A general scheme for inexact reduced gradient (IRG) methods is proposed, where the errors in the gradient approximation automatically adapt with the magnitudes of the exact gradients. The sequences of iterations are shown to obtain stationary accumulation points when different stepsize selections are employe…
For finite-dimensional problems, stochastic approximation methods have long been used to solve stochastic optimization problems. Their application to infinite-dimensional problems is less understood, particularly for nonconvex objectives. This paper presents convergence results for the stochastic proximal gradient method applied to Hilbert spaces, motivated by optimization problems with partial differential equation (PDE) constraints with random inputs and coefficients. We study stochastic algorithms for nonconvex and nonsmooth problems, where the nonsmooth part is convex and the nonconvex par…
The uniform quadratic optimizatin problem (UQ) is a nonconvex quadratic constrained quadratic programming (QCQP) sharing the same Hessian matrix. Based on the second-order cone programming (SOCP) relaxation, we establish a new sufficient condition to guarantee strong duality for (UQ) and then extend it to (QCQP), which not only covers several well-known results in literature but also partially gives answers to a few open questions. For convex constrained nonconvex (UQ), we propose an improved approximation algorithm based on (SOCP). Our approximation bound is dimensional independent. As an app…
We consider a class of nonconvex nonsmooth optimization problems whose objective is the sum of a smooth function and a finite number of nonnegative proper closed possibly nonsmooth functions (whose proximal mappings are easy to compute), some of which are further composed with linear maps. This kind of problems arises naturally in various applications when different regularizers are introduced for inducing simultaneous structures in the solutions. Solving these problems, however, can be challenging because of the coupled nonsmooth functions: the corresponding proximal mapping can be hard to co…
The paper concerns optimal control of discontinuous differential inclusions of the normal cone type governed by a generalized version of the Moreau sweeping process with control functions acting in both nonconvex moving sets and additive perturbations. This is a new class of optimal control problems in comparison with previously considered counterparts where the controlled sweeping sets are described by convex polyhedra. Besides a theoretical interest, a major motivation for our study of such challenging optimal control problems with intrinsic state constraints comes from the application to th…
Many optimization problems arising in high-dimensional statistics decompose naturally into a sum of several terms, where the individual terms are relatively simple but the composite objective function can only be optimized with iterative algorithms. In this paper, we are interested in optimization problems of the form F(Kx) + G(x), where K is a fixed linear transformation, while F and G are functions that may be nonconvex and/or nondifferentiable. In particular, if either of the terms are nonconvex, existing alternating minimization techniques may fail to converge; other types of existing appr…
This paper is devoted to the study, for the first time in the literature, of optimal control problems for sweeping processes governed by integro-differential inclusions of the Volterra type with different classes of control functions acting in nonconvex moving sets, external dynamic perturbations, and integral parts of the sweeping dynamics. We establish the existence of optimal solutions and then obtain necessary optimality conditions for a broad class of local minimizers in such problems. Our approach to deriving necessary optimality conditions is based on the method of discrete approximatio…
This paper revisits the well-studied fixed point problem from a unified viewpoint of mathematical modeling and canonical duality theory, i.e. the original problem is first reformulated as a nonconvex optimization problem, its well-posedness is discussed based on objectivity principle in continuum physics; then the canonical duality theory is applied for solving this problem to obtain not only all fixed points, but also their stability properties. Applications are illustrated by challenging problems governed by nonconvex polynomial, exponential, and logarithmic operators. This paper shows that …
This paper presents two inexact composite gradient methods, one inner accelerated and another doubly accelerated, for solving a class of nonconvex spectral composite optimization problems. More specifically, the objective function for these problems is of the form $f_1 + f_2 + h$ where $f_1$ and $f_2$ are differentiable nonconvex matrix functions with Lipschitz continuous gradients, $h$ is a proper closed convex matrix function, and both $f_2$ and $h$ can be expressed as functions that operate on the singular values of their inputs. The methods essentially use an accelerated composite gradient…
