We show that unless P=NP, there cannot be a polynomial-time algorithm that finds a point within Euclidean distance $c^n$ (for any constant $c \ge 0$) of a local minimizer of an $n$-variate quadratic function over a polytope. This result (even with $c=0$) answers a question of Pardalos and Vavasis that appeared in 1992 on a list of seven open problems in complexity theory for numerical optimization. Our proof technique also implies that the problem of deciding whether a quadratic function has a local minimizer over an (unbounded) polyhedron, and that of deciding if a quartic polynomial has a local minimizer are NP-hard.
We study nonconvex homogeneous quadratically constrained quadratic optimization with one or two constraints, denoted by (QQ1) and (QQ2), respectively. (QQ2) contains (QQ1), trust region subproblem (TRS) and ellipsoid regularized total least squares problem as special cases. It is known that there is a necessary and sufficient optimality condition for the global minimizer of (QQ2). In this paper, we first show that any local minimizer of (QQ1) is globally optimal. Unlike its special case (TRS) with at most one local non-global minimizer, (QQ2) may have infinitely many local non-global minimizer…
Sufficient Conditions for Error Distance Reduction in the (\ell^2)-norm Trust Region between Minimizers of Local Nonconvex Multivariate Quadratic Approximates
We consider two polytopes. The quadratic assignment polytope $QAP(n)$ is the convex hull of the set of tensors $x\otimes x$, $x \in P_n$, where $P_n$ is the set of $n\times n$ permutation matrices. The second polytope is defined as follows. For every permutation of vertices of the complete graph $K_n$ we consider appropriate $\binom{n}{2} \times \binom{n}{2}$ permutation matrix of the edges of $K_n$. The Young polytope $P((n-2,2))$ is the convex hull of all such matrices. In 2009, S. Onn showed that the subgraph isomorphism problem can be reduced to optimization both over $QAP(n)$ and over $P(…
A quadratically constrained quadratic program (QCQP) is an optimization problem in which the objective function is a quadratic function and the feasible region is defined by quadratic constraints. Solving non-convex QCQP to global optimality is a well-known NP-hard problem and a traditional approach is to use convex relaxations and branch-and-bound algorithms. This paper makes a contribution in this direction by showing that the exact convex hull of a general quadratic equation intersected with any bounded polyhedron is second-order cone representable. We present a simple constructive proof of…
In this paper, we analyze some theoretical properties of the problem of minimizing a quadratic function with a cubic regularization term, arising in many methods for unconstrained and constrained optimization that have been proposed in the last years. First we show that, given any stationary point that is not a global solution, it is possible to compute, in closed form, a new point with a smaller objective function value. Then, we prove that a global minimizer can be obtained by computing a finite number of stationary points. Finally, we extend these results to the case where stationary condit…
In this paper we consider three minimization problems, namely quadratic, $ρ$-convex and quadratic fractional programing problems. The quadratic problem is considered with quadratic inequality constraints with bounded continuous and discrete mixed variables. The $ρ$-convex problem is considered with $ρ$-convex inequality constraints in mixed variables. The quadratic fractional problem is studied with quadratic fractional constraints in mixed variables. For all three problems we reformulate the problem as a mathematical programming problem and apply standard Karush Kuhn Tucker necessary conditio…
Newton's method for finding an unconstrained minimizer for strictly convex functions, generally speaking, does not converge from any starting point. We introduce and study the damped regularized Newton's method (DRNM). It converges globally for any strictly convex function, which has a minimizer in $R^n$. Locally DRNM converges with a quadratic rate. We characterize the neighborhood of the minimizer, where the quadratic rate occurs. Based on it we estimate the number of DRNM's steps required for finding an $\varepsilon$- approximation for the minimizer.
Generalized trust-region subproblem (GT) is a nonconvex quadratic optimization with a single quadratic constraint. It reduces to the classical trust-region subproblem (T) if the constraint set is a Euclidean ball. (GT) is polynomially solvable based on its inherent hidden convexity. In this paper, we study local minimizers of (GT). Unlike (T) with at most one local nonglobal minimizer, we can prove that two-dimensional (GT) has at most two local nonglobal minimizers, which are shown by example to be attainable. The main contribution of this paper is to prove that, at any local nonglobal minimi…
Hypergraph matching is a fundamental problem in computer vision. Mathematically speaking, it maximizes a polynomial objective function, subject to assignment constraints. In this paper, we reformulate the hypergraph matching problem as a sparse constrained tensor optimization problem. The optimality conditions are characterized based on the sparse constrained optimization theory. By dropping the sparsity constraint, we show that the resulting relaxation problem can recover the global minimizer of the original problem. The critical step in solving the original problem is to identify the locatio…
This paper is devoted to the study of second order optimality conditions for strong local minimizers in the frameworks of unconstrained and constrained optimization problems in finite dimensions via subgradient graphical derivative. We prove that the positive definiteness of the subgradient graphical derivative of an extended-real-valued lower semicontinuous proper function at a proximal stationary point is sufficient for the quadratic growth condition. It is also a necessary condition for the latter property when the function is either subdifferentially continuous, prox-regular, twice epi-dif…
The condition number of a smooth convex function, namely the ratio of its smoothness to strong convexity constants, is closely tied to fundamental properties of the function. In particular, the condition number of a quadratic convex function is precisely the square of the diameter-to-width ratio of a canonical ellipsoid associated to the function. Furthermore, the condition number of a function bounds the linear rate of convergence of the gradient descent algorithm for unconstrained minimization. We propose a condition number of a smooth convex function relative to a reference polytope. This r…
A counterexample to uniqueness of global minimizers of semilinear optimal control problems is given. The lack of uniqueness occurs for a special choice of the state-target in the cost functional. Our arguments show also that, for some state-targets, there exist local minimizers, which are not global. When this occurs, gradient-type algorithms may be trapped by the local minimizers, thus missing the global ones. Furthermore, the issue of convexity of quadratic functional in optimal control is analyzed in an abstract setting. As a Corollary of the nonuniqueness of the minimizers, a nonuniqueness…
Kepler (1619) and Croft (1980) have considered largest homothetic copies of one regular polytope contained in another regular polytope. For arbitrary pairs of polytopes we propose to model this as a quadratically constrained optimization problem. These problems can then be solved numerically; in case the optimal solutions are algebraic, exact optima can be recovered by solving systems of equations to very high precision and then using integer relation algorithms. Based on this approach, we complete Croft's solution to the problem concerning maximal inclusions of regular three-dimensional polyh…
Let $T>0$ fixed. We consider the optimal control problem for analytic affine systems: $\ds{\dot{x}=f\_0(x)+\sum\_{i=1}^m u\_if\_i(x)}$, with a cost of the form: $\ds{C(u)=\int\_0^T \sum\_{i=1}^m u\_i^2(t)dt}$. For this kind of systems we prove that if there are no minimizing abnormal extremals then the value function $S$ is subanalytic. Secondly we prove that if there exists an abnormal minimizer of corank 1 then the set of end-points of minimizers at cost fixed is tangent to a given hyperplane. We illustrate this situation in sub-Riemannian geometry.
We derive an explicit formula, as well as an efficient procedure, for constructing a generalized Jacobian for the projector of a given square matrix onto the Birkhoff polytope, i.e., the set of doubly stochastic matrices. To guarantee the high efficiency of our procedure, a semismooth Newton method for solving the dual of the projection problem is proposed and efficiently implemented. Extensive numerical experiments are presented to demonstrate the merits and effectiveness of our method by comparing its performance against other powerful solvers such as the commercial software Gurobi and the a…
It is known that the gradient descent algorithm converges linearly when applied to a strongly convex function with Lipschitz gradient. In this case the algorithm's rate of convergence is determined by the condition number of the function. In a similar vein, it has been shown that a variant of the Frank-Wolfe algorithm with away steps converges linearly when applied to a strongly convex function with Lipschitz gradient over a polytope. In a nice extension of the unconstrained case, the algorithm's rate of convergence is determined by the product of the condition number of the function and a cer…
Gradient descent with adaptive stepsize converges (nearly) linearly under fourth-order growth
Wharton Department of Statistics and Data Science, University of Pennsylvania（沃顿统计与数据科学系，宾夕法尼亚大学）；Department of Mathematics, U. Washington（数学系，华盛顿大学）；Edwardson School of Industrial Engineering, Purdue University（工业工程学院，普渡大学）
An alternating direction method of multipliers (ADMM) solver is described for optimal resource allocation problems with separable convex quadratic costs and constraints and linear coupling constraints. We describe a parallel implementation of the solver on a graphics processing unit (GPU) using a bespoke quartic function minimizer. An application to robust optimal energy management in hybrid electric vehicles is described, and the results of numerical simulations comparing the computation times of the parallel GPU implementation with those of an equivalent serial implementation are presented.
This paper studies loss functions for finite sets. For a given finite set $S$, we give sum-of-square type loss functions of minimum degree. When $S$ is the vertex set of a standard simplex, we show such loss functions have no spurious minimizers (i.e., every local minimizer is a global one). Up to transformations, we give similar loss functions without spurious minimizers for general finite sets. When $S$ is approximately given by a sample set $T$, we show how to get loss functions by solving a quadratic optimization problem. Numerical experiments and applications are given to show the efficie…
We consider the task of reconstructing polytopes with fixed facet directions from finitely many support function evaluations. We show that for a fixed simplicial normal fan the least-squares estimate is given by a convex quadratic program. We study the geometry of the solution set and give a combinatorial characterization for the uniqueness of the reconstruction in this case. We provide an algorithm that, under mild assumptions, converges to the unknown input shape as the number of noisy support function evaluations increases. We also discuss limitations of our results if the restriction on th…
In several experimental reports on nonconvex optimization problems in machine learning, stochastic gradient descent (SGD) was observed to prefer minimizers with flat basins in comparison to more deterministic methods, yet there is very little rigorous understanding of this phenomenon. In fact, the lack of such work has led to an unverified, but widely-accepted stochastic mechanism describing why SGD prefers flatter minimizers to sharper minimizers. However, as we demonstrate, the stochastic mechanism fails to explain this phenomenon. Here, we propose an alternative deterministic mechanism that…
This paper is devoted to the general problem of projection onto a polyhedral convex cone generated by a finite set of generators.This problem is reformulated into projection onto the polytope obtained by simple truncation of the original cone. Then it can be solved with just two closely related projections onto the same bounded polytope. This approach's computational performance is conditioned by the crucial tool's efficiency for solving the fundamental problem of finding the least norm element in a convex hull of a given finite set of points. In our numerical experiments, we used for this pur…
We consider maximizing a continuous convex function over the assignment polytope. Such problems arise in Graph Matching (the optimization version of Graph Isomorphism) and Quadratic Assignment problems. In the typical case of maximizing a convex function over a polytope the problem can be solved by using a simplicial algorithm such as Tuy's method or the Falk-Hoffman method, but these algorithms require that the underlying polytope be non-degenerate, which is not the case for the assignment polytope. In this note we show how a simple perturbation scheme can be used to create a "surrogate probl…
Technion - Israel Institute of Technology（以色列理工学院）
In recent years it was proved that simple modifications of the classical Frank-Wolfe algorithm (aka conditional gradient algorithm) for smooth convex minimization over convex and compact polytopes, converge with linear rate, assuming the objective function has the quadratic growth property. However, the rate of these methods depends explicitly on the dimension of the problem which cannot explain their empirical success for large scale problems. In this paper we first demonstrate that already for very simple problems and even when the optimal solution lies on a low-dimensional face of the polyt…
The past decade has witnessed a successful application of deep learning to solving many challenging problems in machine learning and artificial intelligence. However, the loss functions of deep neural networks (especially nonlinear networks) are still far from being well understood from a theoretical aspect. In this paper, we enrich the current understanding of the landscape of the square loss functions for three types of neural networks. Specifically, when the parameter matrices are square, we provide an explicit characterization of the global minimizers for linear networks, linear residual n…
Recently, Sinkhorn's algorithm was applied for approximately solving linear programs emerging from optimal transport very efficiently. This was accomplished by formulating a regularized version of the linear program as Bregman projection problem onto the polytope of doubly-stochastic matrices, and then computing the projection using the efficient Sinkhorn algorithm, which is based on alternating closed-form Bregman projections on the larger polytopes of row-stochastic and column-stochastic matrices. In this paper we suggest a generalization of this algorithm for solving a well-known lifted lin…
This paper mainly studies the quadratic growth and the strong metric subregularity of the subdifferential of a function that can be represented as the sum of a function twice differentiable in the extended sense and a subdifferentially continuous, prox-regular, twice epi-differentiable function. For such a function, which is not necessarily prox-regular, it is shown that the quadratic growth, the strong metric subregularity of the subdifferential at a local minimizer, and the positive definiteness of the subgradient graphical derivative at a stationary point are equivalent. In addition, other …
An open question contributed by Yu. Orlov to a recently published volume "Unsolved Problems in Mathematical Systems and Control Theory", V.D. Blondel, A. Megretski (eds), Princeton Univ. Press, 2004, concerns regularization of optimal control-affine problems. These noncoercive problems in general admit 'cheap (generalized) controls' as minimizers; it has been questioned whether and under what conditions infima of the regularized problems converge to the infimum of the original problem. Starting with a study of this question we show by simple functional-theoretic reasoning that it admits, in ge…
Boolean quadratic optimization problems occur in a number of applications. Their mixed integer-continuous nature is challenging, since it is inherently NP-hard. For this motivation, semidefinite programming relaxations (SDR's) are proposed in the literature to approximate the solution, which recasts the problem into convex optimization. Nevertheless, SDR's do not guarantee the extraction of the correct binary minimizer. In this paper, we present a novel approach to enhance the binary solution recovery. The key of the proposed method is the exploitation of known information on the eigenvalues o…
The composite $L_q~(0<q<1)$ minimization problem over a general polyhedron has received various applications in machine learning, wireless communications, image restoration, signal reconstruction, etc. This paper aims to provide a theoretical study on this problem. Firstly, we show that for any fixed $0<q<1$, finding the global minimizer of the problem, even its unconstrained counterpart, is strongly NP-hard. Secondly, we derive Karush-Kuhn-Tucker (KKT) optimality conditions for local minimizers of the problem. Thirdly, we propose a smoothing sequential quadratic programming framework for solv…
A second-order cone representable class of nonconvex quadratic programs
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院 H. 米尔顿·斯图尔特工业与系统工程学）；Department of Industrial and Systems Engineering, Lehigh University（利哈伊大学工业与系统工程学系）
The constrained linear quadratic regulation problem is solved by a continuous piecewise affine function on a set of state space polytopes. It is an obvious question whether this solution can be built up iteratively by increasing the horizon, i.e., by extending the classical backward dynamic programming solution for the unconstrained case to the constrained case. Unfortunately, however, the piecewise affine solution for horizon N is in general not contained in the piecewise affine law for horizon N + 1. We show that a backward dynamic programming does, in contrast, result in a useful structure …
We study a linear quadratic optimal control problem with stochastic coefficients and a terminal state constraint, which may be in force merely on a set with positive, but not necessarily full probability. Under such a partial terminal constraint, the usual approach via a coupled system of a backward stochastic Riccati equation and a linear backward equation breaks down. As a remedy, we introduce a family of auxiliary problems parametrized by the supersolutions to this Riccati equation alone. The target functional of these problems dominates the original constrained one and allows for an explic…
We present a numerical method for the minimization of objectives that are augmented with large quadratic penalties of overdetermined inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of equality constrained optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM) for solving this class of problems. However, if the equality constraints of the discretization are inconsistent, then ALM might not converge to a point that minimizes the unconstrained b…
We study the convex hull of the graph of a quadratic function $f(\mathbf{x})=\sum_{ij\in E}x_ix_j$, where the sum is over the edge set of a graph $G$ with vertex set $\{1,\dots,n\}$. Using an approach proposed by Gupte et al. (Discrete Optimization $\textbf{36}$, 2020, 100569), we investigate minimal extended formulations using additional variables $y_{ij}$, $1\leq i<j\leq n$, representing the products $x_ix_j$. The basic idea is to identify a set of facets of the Boolean Quadric Polytope which is sufficient for characterizing the convex hull for the given graph. Our main results are extended …
A simple inverse problem for the wave equation requires determination of both the wave velocity in a homogenous acoustic material and the transient waveform of an isotropic point radiator, given the time history of the wavefield at a remote point in space. The duration (support) of the source waveform and the source-to-receiver distance are assumed known. A least squares formulation of this problem exhibits the "cycle-skipping" behaviour observed in field scale problems of this type, with many local minima differing greatly from the global minimizer. An extended formulation, dropping the suppo…
This paper characterizes the solution to a finite horizon min-max optimal control problem where the system is linear and discrete-time with control and state constraints, and the cost quadratic; the disturbance is negatively costed, as in the standard H-infinity problem, and is constrained. The cost is minimized over control policies and maximized over disturbance sequences so that the solution yields a feedback control. It is shown that the value function is piecewise quadratic and the optimal control policy piecewise affine, being quadratic and affine, respectively, in polytopes that partiti…
We present a numerical method for the minimization of constrained optimization problems where the objective is augmented with large quadratic penalties of inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM). However, if the equality constraints are inconsistent, then ALM might not converge to a point that minimizes the bias of the objective and penalty term. Therefore, we present a modific…
Two-view triangulation is a problem of minimizing a quadratic polynomial under an equality constraint. We derive a polynomial that encodes the local minimizers of this problem using the theory of Lagrange multipliers. This offers a simpler derivation of the critical points that are given in Hartley-Sturm [6].
A procedure for generating non-differentiable, continuously differentiable, and twice continuously differentiable classes of test functions for multiextremal multidimensional box-constrained global optimization and a corresponding package of C subroutines are presented. Each test class consists of 100 functions. Test functions are generated by defining a convex quadratic function systematically distorted by polynomials in order to introduce local minima. To determine a class, the user defines the following parameters: (i) problem dimension, (ii) number of local minima, (iii) value of the globa…
