Inspired by alternating direction method of multipliers and the idea of operator splitting, we propose a efficient algorithm for solving large-scale quadratically constrained basis pursuit. Experimental results show that the proposed algorithm can achieve 50~~100 times speedup when compared with the baseline interior point algorithm implemented in CVX.
Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. …
Efficient Algorithms for Robust and Stable Principal Component Pursuit Problems
鲁棒主成分分析(Robust Principal Component Analysis,RPCA)是从一组含有稀疏粗差的观测值中恢复低秩矩阵的方法,在计算机视觉、图像处理和网络数据排序等领域有着广泛的应用。在一定条件下,NP难RPCA问题的解可以通过求解一个凸优化问题,即鲁棒主元追踪(RPCP)来得到。此外,如果观察到的数据矩阵也被破坏的密集噪声矩阵,除了粗稀疏误差,然后稳定的主成分追踪(SPCP)问题的解决,以恢复低秩矩阵。在本文中,我们开发了有效的算法与可证明的迭代复杂性界解决RPCP和SPCP。对数百万变量和约束条件的问题(如从监控视频中提取前景,从人脸图像中去除阴影和镜面反射以及从严重损坏的数据中去除视频去噪)的数值结果表明,我们的算法在精度和速度方面与当前最先进的RPCP和SPCP求解器具有竞争力。
Hard-thresholding-based algorithms have seen various advantages for sparse optimization in controlling the sparsity and allowing for fast computation. Recent research shows that when techniques of the Newton-type methods are integrated, their numerical performance can be improved surprisingly. This paper develops a gradient projection Newton pursuit algorithm that mainly adopts the hard-thresholding operator and employs the Newton pursuit only when certain conditions are satisfied. The proposed algorithm is capable of converging globally and quadratically under the standard assumptions. When i…
Recent methods for learning a linear subspace from data corrupted by outliers are based on convex $\ell_1$ and nuclear norm optimization and require the dimension of the subspace and the number of outliers to be sufficiently small. In sharp contrast, the recently proposed Dual Principal Component Pursuit (DPCP) method can provably handle subspaces of high dimension by solving a non-convex $\ell_1$ optimization problem on the sphere. However, its geometric analysis is based on quantities that are difficult to interpret and are not amenable to statistical analysis. In this paper we provide a ref…
The aim of structured optimization is to assemble a solution, using a given set of (possibly uncountably infinite) atoms, to fit a model to data. A two-stage algorithm based on gauge duality and bundle method is proposed. The first stage discovers the optimal atomic support for the primal problem by solving a sequence of approximations of the dual problem using a bundle-type method. The second stage recovers the approximate primal solution using the atoms discovered in the first stage. The overall approach leads to implementable and efficient algorithms for large problems.
Semidefinite programs (SDP) are important in learning and combinatorial optimization with numerous applications. In pursuit of low-rank solutions and low complexity algorithms, we consider the Burer--Monteiro factorization approach for solving SDPs. We show that all approximate local optima are global optima for the penalty formulation of appropriately rank-constrained SDPs as long as the number of constraints scales sub-quadratically with the desired rank of the optimal solution. Our result is based on a simple penalty function formulation of the rank-constrained SDP along with a smoothed ana…
Instituto de Sistemas e Robótica (ISR), Instituto Superior Técnico (IST), Technical University of Lisbon, Portugal(里斯本理工大学高等技术学院系统机器人研究所);Department of Electrical and Computer Engineering at Carnegie Mellon University, USA(卡内基梅隆大学电气与计算机工程系);Department of Computer Science at ETH Zurich, Switzerland(苏黎世联邦理工学院计算机科学系)
We introduce a new technique for solving uni-parametric versions of linear programs, convex quadratic programs, and linear complementarity problems in which a single parameter is permitted to be present in any of the input data. We demonstrate the use of our method on a small, motivating example and present the results of a small number of computational tests demonstrating its utility for larger scale problems.
We develop procedures, based on minimization of the composition $f(x) = h(c(x))$ of a convex function $h$ and smooth function $c$, for solving random collections of quadratic equalities, applying our methodology to phase retrieval problems. We show that the prox-linear algorithm we develop can solve phase retrieval problems---even with adversarially faulty measurements---with high probability as soon as the number of measurements $m$ is a constant factor larger than the dimension $n$ of the signal to be recovered. The algorithm requires essentially no tuning---it consists of solving a sequence…
A new algorithm for solving the solution of the linear-quadratic optimization problem (LQP) with unseparated boundary conditions in the continuous case is given. Using the properties of symmetry of the corresponding Hamiltonian matrix, the Euler-Lagrange equations, it is shown that linear algebraic equations for determining the missing initial data of the system being solved have a symmetric principal matrix. The results are illustrated by the example of LQP optimization (stationary case) with minimal control actions.
The quadratic shortest path problem is the problem of finding a path in a directed graph such that the sum of interaction costs over all pairs of arcs on the path is minimized. We derive several semidefinite programming relaxations for the quadratic shortest path problem with a matrix variable of order $m+1$, where $m$ is the number of arcs in the graph. We use the alternating direction method of multipliers to solve the semidefinite programming relaxations. Numerical results show that our bounds are currently the strongest bounds for the quadratic shortest path problem. We also present comput…
Recovering a low-rank matrix from highly corrupted measurements arises in compressed sensing of structured high-dimensional signals (e.g., videos and hyperspectral images among others). Robust principal component analysis (RPCA), solved via principal component pursuit (PCP), recovers a low-rank matrix from sparse corruptions that are of unknown value and support by decomposing the observation matrix into two terms: a low-rank matrix and a sparse one, accounting for sparse noise and outliers. In the more general setting, where only a fraction of the data matrix has been observed, low-rank matri…
An Alternating Direction Method with Increasing Penalty for Stable Principal Component Pursuit
稳定主元追踪(Stable Principal Component Pursuit,SPCP)是一个非光滑的凸优化问题,它的解使得即使只有一小部分数据是可观测的,也能可靠地恢复被密集噪声矩阵破坏的数据矩阵的低秩和稀疏分量。本文提出了一种求解SPCP的新算法。该算法是一个修改的交替方向法的乘法器(ADMM),我们使用一个递增序列的罚参数,而不是一个固定的罚款。该算法是基于部分变量分裂和工作直接与非光滑目标函数。我们证明了原始序列和对偶序列在罚参数序列上在较弱的条件下收敛。据我们所知,这是第一个收敛结果的可变罚款ADMM时,罚款是没有界的,目标函数是非光滑的,它的次微分不是一致有界的。使用部分变量分裂和采用递增序列的惩罚乘数,一起,显着减少所需的迭代次数,以实现实际的可行性。我们的初步计算测试表明,该算法在实践中非常好,并优于ASALM,一个国家的最先进的ADMM算法的SPCP问题与一个常数罚参数。
In this paper, we propose a two-phase algorithm for solving continuous rank-one quadratic knapsack problems (R1QKP). In particular, we study the solution structure of the problem without the knapsack constraint. We propose an $O(n\log n)$ algorithm in this case. We then use the solution structure to propose an $O(n^2\log n)$ algorithm that finds an interval containing the optimal value of the Lagrangian dual of R1QKP. In the second phase, we solve the restricted Lagrangian dual problem using a traditional single-variable optimization method. We perform a computational test on random instances …
Ordinary differential equations that model technical systems often contain states, that are considered dangerous for the system. A trajectory that reaches such a state usually indicates a flaw in the design. In this paper, we present and study the properties of an algorithm for finding such trajectories. That is, for a given ordinary differential equation, the algorithm finds a trajectory that originates in one set of states and reaches another one. The algorithm is based on sequential quadratic programming applied to a regularized optimization problem obtained by multiple shooting.
This paper analyzes the iteration-complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. More specifically, the objective function is of the form $f + h$ where $f$ is a differentiable function whose gradient is Lipschitz continuous and $h$ is a closed convex function with bounded domain. The method, basically, consists of applying an accelerated inexact proximal point method for solving approximately a sequence of quadratic penalized subproblems associated to the linearly constrained problem. Each subproblem of…
Quantum adiabatic evolution is perceived as useful for binary quadratic programming problems that are a priori unconstrained. For constrained problems, it is a common practice to relax linear equality constraints as penalty terms in the objective function. However, there has not yet been proposed a method for efficiently dealing with inequality constraints using the quantum adiabatic approach. In this paper, we give a method for solving the Lagrangian dual of a binary quadratic programming (BQP) problem in the presence of inequality constraints and employ this procedure within a branch-and-bou…
Texas A&M University(德克萨斯A&M大学);Cainiao Network(菜鸟网络);University of California, Davis(加州大学戴维斯分校);The Chinese University of Hong Kong(香港中文大学)
We consider the problem of maximizing the $\ell_1$ norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms for tackling this problem. Specifically, our contribution is threefold. First, we present a manifold proximal point …
Mathematical programs with complementarity constraints are notoriously difficult to solve due to their nonconvexity and lack of constraint qualifications in every feasible point. This work focuses on the subclass of quadratic programs with linear complementarity constraints. A novel approach to solving a penalty reformulation using sequential convex programming and a homotopy on the penalty parameter is introduced. Linearizing the necessarily nonconvex penalty function yields convex quadratic subproblems, which have a constant Hessian matrix throughout all iterates. This allows solution comput…
- In this paper we introduce a new method to solve fixed-delay optimal control problems which exploits numerical homotopy procedures. It is known that solving this kind of problems via indirect methods is complex and computationally demanding because their implementation is faced with two difficulties: the extremal equations are of mixed type, and besides, the shooting method has to be carefully initialized. Here, starting from the solution of the non-delayed version of the optimal control problem, the delay is introduced by numerical homotopy methods. Convergence results, which ensure the eff…
Communication Science Laboratories, NTT, Inc.(NTT通信科学实验室);The University of Osaka(大阪大学);The University of Tokyo(东京大学);RIKEN AIP(日本研究机构AIP)
We propose a data-driven framework for efficiently solving quadratic programming (QP) problems by reducing the number of variables in high-dimensional QPs using instance-specific projection. A graph neural network-based model is designed to generate projections tailored to each QP instance, enabling us to produce high-quality solutions even for previously unseen problems. The model is trained on heterogeneous QPs to minimize the expected objective value evaluated on the projected solutions. This is formulated as a bilevel optimization problem; the inner optimization solves the QP under a given…
Guangdong Ocean University(广东海洋大学);Southeast University(东南大学);University of Leicester(莱斯特大学)
The time-varying quadratic miniaturization (TVQM) problem, as a hotspot currently, urgently demands a more reliable and faster--solving model. To this end, a novel adaptive coefficient constructs framework is presented and realized to improve the performance of the solution model, leading to the adaptive zeroing-type neural dynamics (AZTND) model. Then the AZTND model is applied to solve the TVQM problem. The adaptive coefficients can adjust the step size of the model online so that the solution model converges faster. At the same time, the integration term develops to enhance the robustness o…
A stochastic linear quadratic (LQ) optimal control problem with a pointwise linear equality constraint on the terminal state is considered. A strong Lagrangian duality theorem is proved under a uniform convexity condition on the cost functional and a surjectivity condition on the linear constraint mapping. Based on the Lagrangian duality, two approaches are proposed to solve the constrained stochastic LQ problem. First, a theoretical method is given to construct the closed-form solution by the strong duality. Second, an iterative algorithm, called augmented Lagrangian method (ALM), is proposed…
We present a coordinate ascent method for a class of semidefinite programming problems that arise in non-convex quadratic integer optimization. These semidefinite programs are characterized by a small total number of active constraints and by low-rank constraint matrices. We exploit this special structure by solving the dual problem, using a barrier method in combination with a coordinate-wise exact line search. The main ingredient of our algorithm is the computationally cheap update at each iteration and an easy computation of the exact step size. Compared to interior point methods, our appro…
We derive Nash equilibria for a class of quadratic multi-leader-follower games using the nonsmooth best response function. To overcome the challenge of nonsmoothness, we pursue a smoothing approach resulting in a reformulation as a smooth Nash equilibrium problem. The existence and uniqueness of solutions are proven for all smoothing parameters. Accumulation points of Nash equilibria exist for a decreasing sequence of these smoothing parameters and we show that these candidates fulfill the conditions of s-stationarity and are Nash equilibria to the multi-leader-follower game. Finally, we propo…
In this paper, we discuss the solution of a Quadratic Eigenvalue Complementarity Problem (QEiCP) by using Difference of Convex (DC) programming approaches. We first show that QEiCP can be represented as dc programming problem. Then we investigate different dc programming formulations of QEiCP and discuss their dc algorithms based on a well-known method -- DCA. A new local dc decomposition is proposed which aims at constructing a better dc decomposition regarding to the specific feature of the target problem in some neighborhoods of the iterates. This new procedure yields faster convergence and…
Quadratic optimization problems (QPs) are ubiquitous, and solution algorithms have matured to a reliable technology. However, the precision of solutions is usually limited due to the underlying floating-point operations. This may cause inconveniences when solutions are used for rigorous reasoning. We contribute on three levels to overcome this issue. First, we present a novel refinement algorithm to solve QPs to arbitrary precision. It iteratively solves refined QPs, assuming a floating-point QP solver oracle. We prove linear convergence of residuals and primal errors. Second, we provide an ef…
We propose a new formulation of quadratic optimization problems. The objective function $F(f(x),g(x))$ is given as composition of a quadratic function $F(z)$ with two $n$-variate quadratic functions $z_1=f(x)$ and $z_2=g(x).$ In addition, it incorporates with a set of linear inequality constraints in $z=(z_1,z_2)^T,$ while having an implicit constraint that $z$ belongs to the joint numerical range of $(f,g).$ The formulation is very general in the sense that it covers quadratic programming with a single quadratic constraint of all types, including the inequality-type, the equality-type, and th…
A Douglas-Rachford Splitting Method for Solving Monotone Variational Inequalities in Linear-quadratic Dynamic Games
Rotterdam School of Management, Erasmus University(鹿特丹管理学院,伊拉斯姆斯大学);Division of Decision and Control Systems, KTH Royal Institute of Technology(决策与控制系,皇家理工学院);Delft Center for Systems and Control, Delft University of Technology(系统与控制中心,代尔夫特理工大学)
The alternating direction method of multipliers (ADMM) is a widely used method for solving many convex minimization models arising in signal and image processing. In this paper, we propose an inertial ADMM for solving a two-block separable convex minimization problem with linear equality constraints. This algorithm is obtained by making use of the inertial Douglas-Rachford splitting algorithm to the corresponding dual of the primal problem. We study the convergence analysis of the proposed algorithm in infinite-dimensional Hilbert spaces. Furthermore, we apply the proposed algorithm on the rob…
We present a numerical method for the minimization of objectives that are augmented with large quadratic penalties of overdetermined inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of equality constrained optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM) for solving this class of problems. However, if the equality constraints of the discretization are inconsistent, then ALM might not converge to a point that minimizes the unconstrained b…
The solution of a constrained linear-quadratic regulator problem is determined by the set of its optimal active sets. We propose an algorithm that constructs this set of active sets for a desired horizon N from that for horizon N-1. While it is not obvious how to extend the optimal feedback law itself for horizon N-1 to horizon N, a simple relation between the optimal active sets for two successive horizon lengths exists. Specifically, every optimal active set for horizon N is a superset of an optimal active set for horizon N-1 if the constraints are ordered stage by stage. The stagewise treat…
Solving quadratic systems of equations in n variables and m measurements of the form $y_i = |a^T_i x|^2$ , $i = 1, ..., m$ and $x \in R^n$ , which is also known as phase retrieval, is a hard nonconvex problem. In the case of standard Gaussian measurement vectors, the wirtinger flow algorithm Chen and Candes (2015) is an efficient solution. In this paper, we proposed a new form of wirtinger flow and a new spectral initialization method based on this new algorithm. We proved that the new wirtinger flow and initialization method achieve linear sample and computational complexities. We further ext…
We present algorithm MIQCR-CB that is an advancement of method MIQCR~(Billionnet, Elloumi and Lambert, 2012). MIQCR is a method for solving mixed-integer quadratic programs and works in two phases: the first phase determines an equivalent quadratic formulation with a convex objective function by solving a semidefinite problem $(SDP)$, and, in the second phase, the equivalent formulation is solved by a standard solver. As the reformulation relies on the solution of a large-scale semidefinite program, it is not tractable by existing semidefinite solvers, already for medium sized problems. To sur…