We consider optimization problems in which the goal is find a $k$-dimensional subspace of $\mathbb{R}^n$, $k<<n$, which minimizes a convex and smooth loss. Such problems generalize the fundamental task of principal component analysis (PCA) to include robust and sparse counterparts, and logistic PCA for binary data, among others. This problem could be approached either via nonconvex gradient methods with highly-efficient iterations, but for which arguing about fast convergence to a global minimizer is difficult or, via a convex relaxation for which arguing about convergence to a global minimizer is straightforward, but the corresponding methods are often inefficient in high dimensions. In this work we bridge these two approaches under a strict complementarity assumption, which in particular implies that the optimal solution to the convex relaxation is unique and is also the optimal solution to the original nonconvex problem. Our main result is a proof that a natural nonconvex gradient method which is \textit{SVD-free} and requires only a single QR-factorization of an $n\times k$ matrix per iteration, converges locally with a linear rate. We also establish linear convergence results for the nonconvex projected gradient method, and the Frank-Wolfe method when applied to the convex relaxation.
Technion - Israel Institute of Technology（以色列理工学院）
In recent years it was proved that simple modifications of the classical Frank-Wolfe algorithm (aka conditional gradient algorithm) for smooth convex minimization over convex and compact polytopes, converge with linear rate, assuming the objective function has the quadratic growth property. However, the rate of these methods depends explicitly on the dimension of the problem which cannot explain their empirical success for large scale problems. In this paper we first demonstrate that already for very simple problems and even when the optimal solution lies on a low-dimensional face of the polyt…
In this paper, we prove that all H$^+$(Z$^+$)-eigenvalues of each principal sub-tensor of a strictly semi-positive tensor are positive. We define two new constants associated with H$^+$(Z$^+$)eigenvalues of a strictly semi-positive tensor. With the help of these two constants, we establish upper bounds of an important quantity whose positivity is a necessary and sufficient condition for a general tensor to be a strictly semi-positive tensor. The monotonicity and boundedness of such a quantity are established too. Furthermore, we present global error bound analysis for a class of the nonlinear …
This paper studies tensor eigenvalue complementarity problems. Basic properties of standard and complementarity tensor eigenvalues are discussed. We formulate tensor eigenvalue complementarity problems as constrained polynomial optimization. When one tensor is strictly copositive, the complementarity eigenvalues can be computed by solving polynomial optimization with normalization by strict copositivity. When no tensor is strictly copositive, we formulate the tensor eigenvalue complementarity problem equivalently as polynomial optimization by a randomization process. The complementarity eigenv…
The MaxCut SDP is one of the most well-known semidefinite programs, and it has many favorable properties. One of its nicest geometric/duality properties is the fact that the vertices of its feasible region correspond exactly to the cuts of a graph, as proved by Laurent and Poljak in 1995. Recall that a boundary point $x$ of a convex set $C$ is called a vertex of $C$ if the normal cone of $C$ at $x$ is full-dimensional. We study how often strict complementarity holds or fails for the MaxCut SDP when a vertex of the feasible region is optimal, i.e., when the SDP relaxation is tight. While strict…
The tensor complementarity problem $(\q, \mathcal{A})$ is to $$\mbox{ find } \x \in \mathbb{R}^n\mbox{ such that }\x \geq \0, \q + \mathcal{A}\x^{m-1} \geq \0, \mbox{ and }\x^\top (\q + \mathcal{A}\x^{m-1}) = 0.$$ We prove that a real tensor $\mathcal{A}$ is a (strictly) semi-positive tensor if and only if the tensor complementarity problem $(\q, \mathcal{A})$ has a unique solution for $\q>\0$ ($\q\geq\0$), and a symmetric real tensor is a (strictly) semi-positive tensor if and only if it is (strictly) copositive. That is, for a strictly copositive symmetric tensor $\mathcal{A}$, the tensor co…
In this paper, we provide an elementary, geometric, and unified framework to analyze conic programs that we call the strict complementarity approach. This framework allows us to establish error bounds and quantify the sensitivity of the solution. The framework uses three classical ideas from convex geometry and linear algebra: linear regularity of convex sets, facial reduction, and orthogonal decomposition. We show how to use this framework to derive error bounds for linear programming (LP), second order cone programming (SOCP), and semidefinite programming (SDP).
We introduce a Kojima-Megiddo-Mizuno type continuation method for solving tensor complementarity problems. We show that there exists a bounded continuation trajectory when the tensor is strictly semi-positive and any limit point tracing the trajectory gives a solution of the tensor complementarity problem. Moreover, when the tensor is strong strictly semi-positive, tracing the trajectory will converge to the unique solution. Some numerical results are given to illustrate the effectiveness of the method.
We develop a novel variant of the classical Frank-Wolfe algorithm, which we call spectral Frank-Wolfe, for convex optimization over a spectrahedron. The spectral Frank-Wolfe algorithm has a novel ingredient: it computes a few eigenvectors of the gradient and solves a small-scale SDP in each iteration. Such procedure overcomes slow convergence of the classical Frank-Wolfe algorithm due to ignoring eigenvalue coalescence. We demonstrate that strict complementarity of the optimization problem is key to proving linear convergence of various algorithms, such as the spectral Frank-Wolfe algorithm as…
In this paper, one of our main purposes is to prove the boundedness of solution set of tensor complementarity problem with B tensor such that the specific bounds only depend on the structural properties of tensor. To achieve this purpose, firstly, we present that each B tensor is strictly semi-positive and each B$_0$ tensor is semi-positive. Subsequencely, the strictly lower and upper bounds of different operator norms are given for two positively homogeneous operators defined by B tensor. Finally, with the help of the upper bounds of different operator norms, we show the strcitly lower bound …
In recent years several classes of structured matrices are extended to classes of tensors in the context of tensor complementarity problem. The tensor complementarity problem is a class of nonlinear complementarity problem where the involved functions are special polynomials defined by a tensor. Semipositive and strictly semipositive tensors play an important role in the study of the tensor complementarity problem. The article considers some important properties of semipositive tensor. We establish invariance property of semipositive tensor. We prove necessary and sufficient conditions for a t…
In this paper, we first introduce R0-W and SSM-W properties for the set of matrices which is a generalization of R0 and the strictly semimonotone matrix. We then prove some existence results for the extended horizontal linear complementarity problem when the involved matrices have these properties. With an additional condition on the set of matrices, we prove that the SSM-W property is equivalent to the unique solution for the corresponding extended horizontal linear complementarity problems. Finally, we give a necessary and sufficient condition for the connectedness of the solution set of the…
This paper deals with the class of Q-tensors, that is, a Q-tensor is a real tensor $\mathcal{A}$ such that the tensor complementarity problem $(\q, \mathcal{A})$: $$\mbox{ finding } \x \in \mathbb{R}^n\mbox{ such that }\x \geq \0, \q + \mathcal{A}\x^{m-1} \geq \0, \mbox{ and }\x^\top (\q + \mathcal{A}\x^{m-1}) = 0, $$ has a solution for each vector $\q \in \mathbb{R}^n$. Several subclasses of Q-tensors are given: P-tensors, R-tensors, strictly semi-positive tensors and semi-positive R$_0$-tensors. We prove that a nonnegative tensor is a Q-tensor if and only if all of its principal diagonal ent…
We present a new interior-point potential-reduction algorithm for solving monotone linear complementarity problems (LCPs) that have a particular special structure: their matrix $M\in{\mathbb R}^{n\times n}$ can be decomposed as $M=ΦU + Π_0$, where the rank of $Φ$ is $k<n$, and $Π_0$ denotes Euclidean projection onto the nullspace of $Φ^\top$. We call such LCPs projective. Our algorithm solves a monotone projective LCP to relative accuracy $ε$ in $O(\sqrt n \ln(1/ε))$ iterations, with each iteration requiring $O(nk^2)$ flops. This complexity compares favorably with interior-point algorithms for…
The tensor complementarity problem is a specially structured nonlinear complementarity problem, then it has its particular and nice properties other than ones of the classical nonlinear complementarity problem. In this paper, it is proved that a tensor is an S-tensor if and only if the tensor complementarity problem is feasible, and each Q-tensor is an S-tensor. Furthermore, the boundedness of solution set of the tensor complementarity problem is equivalent to the uniqueness of solution for such a problem with zero vector. For the tensor complementarity problem with a strictly semi-positive te…
Properties of solutions of the tensor complementarity problem (TCP) for structured tensors have been investigated in recent literature. In this paper, we make further contributions on this problem. Specifically, we first derive solution existence theorems for TCPs on general cones from the results studied in the nonlinear complementarity problem literature. An interesting byproduct is that conditions (e.g., strict copositivity) of solution existence results for TCPs on the nonnegative cone can be reduced to copositivity, which, to the best of our knowledge, is the weakest requirement in the cu…
Complementarity problems and variational inequalities arise in a wide variety of areas, including machine learning, planning, game theory, and physical simulation. In all of these areas, to handle large-scale problem instances, we need fast approximate solution methods. One promising idea is Galerkin approximation, in which we search for the best answer within the span of a given set of basis functions. Bertsekas proposed one possible Galerkin method for variational inequalities. However, this method can exhibit two problems in practice: its approximation error is worse than might be expected …
Augmented Lagrangian methods for nonlinear semidefinite programming with complementarity constraints
Federal University of São Paulo（圣保罗联邦大学）；State University of Maringá（马林加州立大学）；Federal University of Paraná（巴拉那联邦大学）；Instituto Federal de Santa Catarina (IFSC)（圣卡塔琳娜联邦学院）
We present a short step interior point method for solving a class of nonlinear programming problems with quadratic objective function. Convex quadratic programming problems can be reformulated as problems in this class. The method is shown to have weak polynomial time complexity. A complete proof of the numerical stability of the method is provided. No requirements on feasibility, row-rank of the constraint Jacobian, strict complementarity, or conditioning of the problem are made. Infeasible problems are solved to an optimal interior least-squares solution.
Relaxed constant positive linear dependence constraint qualification (RCPLD) for a system of smooth equalities and inequalities is a constraint qualification that is weaker than the usual constraint qualifications such as Mangasarian Fromovitz constraint qualification and the linear constraint qualification. Moreover RCPLD is known to induce an error bound property. In this paper we extend RCPLD to a very general feasibility system which may include Lipschitz continuous inequality constraints, complementarity constraints and abstract constraints. We show that this RCPLD for the general system …
Linear complementarity problems are a powerful tool for modeling many practically relevant situations such as market equilibria. They also connect many sub-areas of mathematics like game theory, optimization, and matrix theory. Despite their close relation to optimization, the protection of LCPs against uncertainties -- especially in the sense of robust optimization -- is still in its infancy. During the last years, robust LCPs have only been studied using the notions of strict and $Γ$-robustness. Unfortunately, both concepts lead to the problem that the existence of robust solutions cannot be…
We derive sufficient conditions for strict dissipativity for optimal control of linear evolution equations on Hilbert spaces with a cost functional including linear and quadratic terms. We show that strict dissipativity with a particular storage function is equivalent to ellipticity of a Lyapunov-like operator. Further we prove under a spectral decomposition assumption of the underlying generator and an orthogonality condition of the resulting subspaces that this ellipticity property holds under a detectability assumption. We illustrate our result by means of an example involving a heat equati…
On the Local Linear Convergence of Operator Splitting Methods for Conic Programming
University of California San Diego（加州大学圣地亚哥分校）；MIT, Sloan School of Management（麻省理工学院斯隆管理学院）；University of Chicago, Department of Statistics（芝加哥大学统计学系）
Utilizing dual descriptions of the normal cone of convex optimization problems in conic form, we characterize the vertices of semidefinite representations arising from Lovász theta body, generalizations of the elliptope, and related convex sets. Our results generalize vertex characterizations due to Laurent and Poljak from the 1990's. Our approach also leads us to nice characterizations of strict complementarity and to connections with some of the related literature.
Finding a zero of a sum of maximally monotone operators is a fundamental problem in modern optimization and nonsmooth analysis. Assuming that resolvents of the operators are available, this problem can be tackled with the Douglas-Rachford algorithm. However, when dealing with three or more operators, one must work in a product space with as many factors as there are operators. In groundbreaking recent work by Ryu and by Malitsky and Tam, it was shown that the number of factors can be reduced by one. These splitting methods guarantee weak convergence to some solution of the underlying sum probl…
In this paper, we will study some variational properties of the Ky Fan $k$-norm $θ=\|\cdot\|_{(k)}$ of matrices, which are closed related to a class of basic nonlinear optimization problems involving the Ky Fan $k$-norm. In particular, for the basic nonlinear optimization problems, we will introduce the concept of nondegeneracy, strict complementarity and the critical cones associated with the generalized equations. Finally, we present the explicit formulas of the conjugate function of the parabolic second order directional derivative of $θ$, which will be referred to as the sigma term of the …
We propose two basic assumptions, under which the rate of convergence of the augmented Lagrange method for a class of composite optimization problems is estimated. We analyze the rate of local convergence of the augmented Lagrangian method for a nonlinear semidefinite nuclear norm composite optimization problem by verifying these two basic assumptions. Without requiring strict complementarity, we prove that, under the constraint nondegeneracy condition and the strong second order sufficient condition, the rate of convergence is linear and the ratio constant is proportional to 1/c, where c is t…
It is well-known that the central path of semi-definite optimization, unlike linear optimization, has no analytic extension to $μ= 0$ in the absence of the strict complementarity condition. In this paper, we show the existence of a positive integer $ρ$ by which the reparametrization $μ\mapsto μ^ρ$ recovers the analyticity of the central path at $μ= 0$. We investigate the complexity of computing $ρ$ using algorithmic real algebraic geometry and the theory of complex algebraic curves. We prove that the optimal $ρ$ is bounded by $2^{O(m^2+n^2m+n^4)}$, where $n$ is the matrix size and $m$ is the n…
