The problem of minimizing the rank of a symmetric positive semidefinite matrix subject to constraints can be cast equivalently as a semidefinite program with complementarity constraints (SDCMPCC). The formulation requires two positive semidefinite matrices to be complementary. This is a continuous and nonconvex reformulation of the rank minimization problem. We investigate calmness of locally optimal solutions to the SDCMPCC formulation and hence show that any locally optimal solution is a KKT point.
We develop a penalty formulation of the problem. We present calmness results for locally optimal solutions to the penalty formulation. We also develop a proximal alternating linearized minimization (PALM) scheme for the penalty formulation, and investigate the incorporation of a momentum term into the algorithm. Computational results are presented.
The primal-dual active set method is observed to be the limit of a sequence of penalty formulations. Using this perspective, we propose a penalty method that adaptively becomes the active set method as the residual of the iterate decreases. The adaptive penalty method (APM) therewith combines the main advantages of both methods, namely the ease of implementation of penalty methods and the exact imposition of inequality constraints inherent to the active set method. The scheme can be considered a quasi-Newton method in which the Jacobian is approximated using a penalty parameter. This spatially…
Affine matrix rank minimization problem is a fundamental problem with a lot of important applications in many fields. It is well known that this problem is combinatorial and NP-hard in general. In this paper, a continuous promoting low rank non-convex fraction function is studied to replace the rank function in this NP-hard problem. Inspired by our former work in compressed sensing, an iterative singular value thresholding algorithm is proposed to solve the regularization transformed affine matrix rank minimization problem. For different $a>0$, we could get a much better result by adjusting th…
National University of Defense Technology（国防科技大学）；Hunan University（湖南大学）
In this paper, we consider a class of nonconvex problems with linear constraints appearing frequently in the area of image processing. We solve this problem by the penalty method and propose the iteratively reweighted alternating minimization algorithm. To speed up the algorithm, we also apply the continuation strategy to the penalty parameter. A convergence result is proved for the algorithm. Compared with the nonconvex ADMM, the proposed algorithm enjoys both theoretical and computational advantages like weaker convergence requirements and faster speed. Numerical results demonstrate the effi…
In the paper, we study the minimization problem of a non-convex sparsity promoting penalty function $$P_{a}(x)=\sum_{i=1}^{n}p_{a}(x_{i})=\sum_{i=1}^{n}\frac{a|x_{i}|}{1+a|x_{i}|}$$ in compressed sensing, which is called fraction function. Firstly, we discuss the equivalence of $\ell_{0}$ minimization and fraction function minimization. It is proved that there corresponds a constant $a^{**}>0$ such that, whenever $a>a^{**}$, every solution to $(FP_{a})$ also solves $(P_{0})$, that the uniqueness of global minimizer of $(FP_{a})$ and its equivalence to $(P_{0})$ if the sensing matrix $A$ satisf…
Recently, penalties promoting signals that are sparse within and across groups have been proposed. In this letter, we propose a generalization that allows to encode more intricate dependencies within groups. However, this complicates the realization of the threshold function associated with the penalty, which hinders the use of the penalty in energy minimization. We discuss how to sidestep this problem, and demonstrate the use of the modified penalty in an energy minimization formulation for an inverse problem.
We reformulate the zero-norm minimization problem as an equivalent mathematical program with equilibrium constraints and establish that its penalty problem, induced by adding the complementarity constraint to the objective, is exact. Then, by the special structure of the exact penalty problem, we propose a decomposition method that can seek a global optimal solution of the zero-norm minimization problem under the null space condition in [M. A. Khajehnejad et al. IEEE Trans. Signal. Process., 59(2011), pp. 1985-2001] by solving a finite number of weighted $l_1$-norm minimization problems. To ha…
We focus on the minimization of the least square loss function either under a $k$-sparse constraint or with a sparse penalty term. Based on recent results, we reformulate the $\ell_0$ pseudo-norm exactly as a convex minimization problem by introducing an auxiliary variable. We then propose an exact biconvex reformulation of the $\ell_2-\ell_0$ constrained and penalized problems. We give correspondence results between minimizers of the initial function and the reformulated ones. The reformulation is biconvex and the non-convexity is due to a penalty term. These two properties are used to derive…
For the rank regularized minimization problem, we introduce several kinds of stationary points by the problem itself and its equivalent reformulations including the mathematical program with an equilibrium constraint (MPEC), the global exact penalty of the MPEC,the surrogate yielded by eliminating the dual part in the exact penalty. A clear relation chart is established for these stationary points, which guides the user to choose an appropriate reformulation for seeking a low-rank solution. As a byproduct, we also provide a weaker condition for a local minimizer of the MPEC to be the M-station…
An algorithm for computing the Karcher mean of $n$ positive definite matrices is proposed, based on the majorization-minimization (MM) principle. The proposed MM algorithm is parameter-free, does not need to choose step sizes, and has a theoretical guarantee of asymptotic linear convergence.
Many real world practical problems can be formulated as $\ell_{0}$-minimization problems with nonnegativity constraints, which seek the sparsest nonnegative signals to underdetermined linear systems. They have been widely applied in signal and image processing, machine learning, pattern recognition and computer vision. Unfortunately, this $\ell_{0}$-minimization problem with nonnegativity constraint is computational and NP-hard because of the discrete and discontinuous nature of the $\ell_{0}$-norm. In this paper, we replace the $\ell_{0}$-norm with a non-convex fraction function, and study th…
Exact Quadratic Penalty Function for Symplectic Eigenvalue Problem
State Key Laboratory of Scientific and Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院计算数学与科学工程计算国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
The $\ell_1$ norm is the tight convex relaxation for the $\ell_0$ "norm" and has been successfully applied for recovering sparse signals. For problems with fewer samplings, one needs to enhance the sparsity by nonconvex penalties such as $\ell_p$ "norm". As one method for solving $\ell_p$ minimization problems, iteratively reweighted $\ell_1$ minimization updates the weight for each component based on the value of the same component at the previous iteration. It assigns large weights on small components in magnitude and small weights on large components in magnitude. In this paper, we consider…
For the problem of sparse recovery, it is widely accepted that nonconvex minimizations are better than $\ell_1$ penalty in enhancing the sparsity of solution. However, to date, the theory verifying that nonconvex penalties outperform (or are at least as good as) $\ell_1$ minimization in exact, uniform recovery has mostly been limited to separable cases. In this paper, we establish general recovery guarantees through null space conditions for nonconvex, non-separable regularizations, which are slightly less demanding than the standard null space property for $\ell_1$ minimization.
This paper provides the first meaningful documentation and analysis of an established technique which aims to obtain an approximate solution to linear programming problems prior to applying the primal simplex method. The underlying algorithm is a penalty method with naive approximate minimization in each iteration. During initial iterations an approach similar to augmented Lagrangian is used. Later the technique corresponds closely to a classical quadratic penalty method. There is also a discussion of the extent to which it can be used to obtain fast approximate solutions of LP problems, in pa…
The theorem like Pontryagin's maximum principle for multiple integrals is proved. Unlike the usual maximum principle, the maximum should be taken not over all matrices, but only on matrices of rank one. Examples are given.
A regularized minimization model with $l_1$-norm penalty (RP) is introduced for training the autoencoders that belong to a class of two-layer neural networks. We show that the RP can act as an exact penalty model which shares the same global minimizers, local minimizers, and d(irectional)-stationary points with the original regularized model under mild conditions. We construct a bounded box region that contains at least one global minimizer of the RP, and propose a linearly constrained regularized minimization model with $l_1$-norm penalty (LRP) for training autoencoders. A smoothing proximal …
We present a numerical method for the minimization of objectives that are augmented with large quadratic penalties of overdetermined inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of equality constrained optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM) for solving this class of problems. However, if the equality constraints of the discretization are inconsistent, then ALM might not converge to a point that minimizes the unconstrained b…
In recent years, the Deep Learning Alternating Minimization (DLAM), which is actually the alternating minimization applied to the penalty form of the deep neutral networks training, has been developed as an alternative algorithm to overcome several drawbacks of Stochastic Gradient Descent (SGD) algorithms. This work develops an improved DLAM by the well-known inertial technique, namely iPDLAM, which predicts a point by linearization of current and last iterates. To obtain further training speed, we apply a warm-up technique to the penalty parameter, that is, starting with a small initial one a…
Symplectic eigenvalues are conventionally defined for symmetric positive-definite matrices via Williamson's diagonal form. Many properties of standard eigenvalues, including the trace minimization theorem, are extended to the case of symplectic eigenvalues. In this note, we will generalize Williamson's diagonal form for symmetric positive-definite matrices to the case of symmetric positive-semidefinite matrices, which allows us to define symplectic eigenvalues, and prove the trace minimization theorem in the new setting.
Common computational problems, such as parameter estimation in dynamic models and PDE constrained optimization, require data fitting over a set of auxiliary parameters subject to physical constraints over an underlying state. Naive quadratically penalized formulations, commonly used in practice, suffer from inherent ill-conditioning. We show that surprisingly the partial minimization technique regularizes the problem, making it well-conditioned. This viewpoint sheds new light on variable projection techniques, as well as the penalty method for PDE constrained optimization, and motivates robust…
As a conclusion in classical linear algebra, an underdetermined linear equations usually have an infinite number of solutions. The sparest one among these solutions is significant in many applications. This problem can be modeled as the $l_0$-minimization, However, to find the sparsest solution of an underdetermined linear equations is NP-hard. Therefore, an important approach to solve the following $l_p$-minimization ($0<p\leq1$), The purpose of this problem is to find a $p$-norm minimization solution $(0<p\leq1)$ instead of the sparest one. In order to study the equivalence relationship betw…
We consider the efficient minimization of a nonlinear, strictly convex functional with $\ell_1$-penalty term. Such minimization problems appear in a wide range of applications like Tikhonov regularization of (non)linear inverse problems with sparsity constraints. In (2015 Inverse Problems (31) 025005), a globalized Bouligand-semismooth Newton method was presented for $\ell_1$-Tikhonov regularization of linear inverse problems. Nevertheless, a technical assumption on the accumulation point of the sequence of iterates was necessary to prove global convergence. Here, we generalize this method to …
We consider the estimation of the state transition matrix in vector autoregressive models, when time sequence data is limited but nonsequence steady-state data is abundant. To leverage both sources of data, we formulate the least squares minimization problem regularized by a Lyapunov penalty. We impose cardinality or rank constraints to reduce the complexity of the autoregressive model. We solve the resulting nonconvex, nonsmooth problem by using the proximal alternating linearization method (PALM). We show that PALM is globally convergent to a critical point and that the estimation error mono…
PhaseLift, proposed by E.J. Candès et al., is one convex relaxation approach for phase retrieval. The relaxation enlarges the solution set from rank one matrices to positive semidefinite matrices. In this paper, a relaxation is employed to nonconvex alternating minimization methods to recover the rank-one matrices. A generic measurement matrix can be standardized to a matrix consisting of orthonormal columns. To recover the rank-one matrix, the standardized frames are used to select the matrix with the maximal leading eigenvalue among the rank-$r$ matrices. Empirical studies are conducted to v…
A Convergent ADMM Algorithm for Grain Boundary Energy Minimization
The Hong Kong University of Science and Technology（香港科技大学）；Shenzhen University（深圳大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（港深创新研究院）
We present a numerical method for the minimization of constrained optimization problems where the objective is augmented with large quadratic penalties of inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM). However, if the equality constraints are inconsistent, then ALM might not converge to a point that minimizes the bias of the objective and penalty term. Therefore, we present a modific…
In this paper, we consider the problem of minimizing a smooth objective over multiple rank constraints on Hankel-structured matrices. This kind of problems arises in system identification, system theory and signal processing, where the rank constraints are typically "hard constraints". To solve these problems, we propose a hybrid penalty method that combines a penalty method with a post-processing scheme. Specifically, we solve the penalty subproblems until the penalty parameter reaches a given threshold, and then switch to a local alternating "pseudo-projection'' method to further reduce cons…
In this two-part study we develop a unified approach to the analysis of the global exactness of various penalty and augmented Lagrangian functions for finite-dimensional constrained optimization problems. This approach allows one to verify in a simple and straightforward manner whether a given penalty/augmented Lagrangian function is exact, i.e. whether the problem of unconstrained minimization of this function is equivalent (in some sense) to the original constrained problem, provided the penalty parameter is sufficiently large. Our approach is based on the so-called localization principle th…
We propose NAMA (Newton-type Alternating Minimization Algorithm) for solving structured nonsmooth convex optimization problems where the sum of two functions is to be minimized, one being strongly convex and the other composed with a linear mapping. The proposed algorithm is a line-search method over a continuous, real-valued, exact penalty function for the corresponding dual problem, which is computed by evaluating the augmented Lagrangian at the primal points obtained by alternating minimizations. As a consequence, NAMA relies on exactly the same computations as the classical alternating min…
Sparsity constrained minimization captures a wide spectrum of applications in both machine learning and signal processing. This class of problems is difficult to solve since it is NP-hard and existing solutions are primarily based on Iterative Hard Thresholding (IHT). In this paper, we consider a class of continuous optimization techniques based on Mathematical Programs with Equilibrium Constraints (MPECs) to solve general sparsity constrained problems. Specifically, we reformulate the problem as an equivalent biconvex MPEC, which we can solve using an exact penalty method or an alternating di…
In this paper, we present a semi-proximal alternating direction method of multipliers (ADMM) for solving $3$-block separable convex minimization problems with the second block in the objective being a strongly convex function and one coupled linear equation constraint. By choosing the semi-proximal terms properly, we establish the global convergence of the proposed semi-proximal ADMM for the step-length $τ\in (0, (1+\sqrt{5})/2)$ and the penalty parameter $σ\in (0, +\infty)$. In particular, if $σ>0$ is smaller than a certain threshold and the first and third linear operators in the linear equa…
Consider the regularized sparse minimization problem, which involves empirical sums of loss functions for $n$ data points (each of dimension $d$) and a nonconvex sparsity penalty. We prove that finding an $\mathcal{O}(n^{c_1}d^{c_2})$-optimal solution to the regularized sparse optimization problem is strongly NP-hard for any $c_1, c_2\in [0,1)$ such that $c_1+c_2<1$. The result applies to a broad class of loss functions and sparse penalty functions. It suggests that one cannot even approximately solve the sparse optimization problem in polynomial time, unless P $=$ NP.
We study quadratic optimization with indicator variables and an M-matrix, i.e., a PSD matrix with non-positive off-diagonal entries, which arises directly in image segmentation and portfolio optimization with transaction costs, as well as a substructure of general quadratic optimization problems. We prove, under mild assumptions, that the minimization problem is solvable in polynomial time by showing its equivalence to a submodular minimization problem. To strengthen the formulation, we decompose the quadratic function into a sum of simple quadratic functions with at most two indicator variabl…
We develop two new proximal alternating penalty algorithms to solve a wide range class of constrained convex optimization problems. Our approach mainly relies on a novel combination of the classical quadratic penalty, alternating minimization, Nesterov's acceleration, and adaptive strategy for parameters. The first algorithm is designed to solve generic and possibly nonsmooth constrained convex problems without requiring any Lipschitz gradient continuity or strong convexity, while achieving the best-known $O(\frac{1}{k})$-convergence rate in a non-ergodic sense, where $k$ is the iteration coun…
Low-rank representation (LRR) is an effective method for subspace clustering and has found wide applications in computer vision and machine learning. The existing LRR solver is based on the alternating direction method (ADM). It suffers from $O(n^3)$ computation complexity due to the matrix-matrix multiplications and matrix inversions, even if partial SVD is used. Moreover, introducing auxiliary variables also slows down the convergence. Such a heavy computation load prevents LRR from large scale applications. In this paper, we generalize ADM by linearizing the quadratic penalty term and allow…
We consider the minimization of a convex objective function subject to the set of minima of another convex function, under the assumption that both functions are twice continuously differentiable. We approach this optimization problem from a continuous perspective by means of a second order dynamical system with Hessian-driven damping and a penalty term corresponding to the constrained function. By constructing appropriate energy functionals, we prove weak convergence of the trajectories generated by this differential equation to a minimizer of the optimization problem as well as convergence f…
In compressed sensing problems, $\ell_1$ minimization or Basis Pursuit was known to have the best provable phase transition performance of recoverable sparsity among polynomial-time algorithms. It is of great theoretical and practical interest to find alternative polynomial-time algorithms which perform better than $\ell_1$ minimization. \cite{Icassp reweighted l_1}, \cite{Isit reweighted l_1}, \cite{XuScaingLaw} and \cite{iterativereweightedjournal} have shown that a two-stage re-weighted $\ell_1$ minimization algorithm can boost the phase transition performance for signals whose nonzero elem…
Distributionally robust optimization (DRO) is a widely used framework for optimizing objective functionals in the presence of both randomness and model-form uncertainty. A key step in the practical solution of many DRO problems is a tractable reformulation of the optimization over the chosen model ambiguity set, which is generally infinite dimensional. Previous works have solved this problem in the case where the objective functional is an expected value. In this paper we study objective functionals that are the sum of an expected value and a variance penalty term. We prove that the correspond…
We propose a new first-order optimisation algorithm to solve high-dimensional non-smooth composite minimisation problems. Typical examples of such problems have an objective that decomposes into a non-smooth empirical risk part and a non-smooth regularisation penalty. The proposed algorithm, called Semi-Proximal Mirror-Prox, leverages the Fenchel-type representation of one part of the objective while handling the other part of the objective via linear minimization over the domain. The algorithm stands in contrast with more classical proximal gradient algorithms with smoothing, which require th…
Matrix decomposition is ubiquitous and has applications in various fields like speech processing, data mining and image processing to name a few. Under matrix decomposition, nonnegative matrix factorization is used to decompose a nonnegative matrix into a product of two nonnegative matrices which gives some meaningful interpretation of the data. Thus, nonnegative matrix factorization has an edge over the other decomposition techniques. In this paper, we propose two novel iterative algorithms based on Majorization Minimization (MM)-in which we formulate a novel upper bound and minimize it to ge…
We propose a convex formulation of the fused lasso signal approximation problem consisting of non-convex penalty functions. The fused lasso signal model aims to estimate a sparse piecewise constant signal from a noisy observation. Originally, the $\ell_1$ norm was used as a sparsity-inducing convex penalty function for the fused lasso signal approximation problem. However, the $\ell_1$ norm underestimates signal values. Non-convex sparsity-inducing penalty functions better estimate signal values. In this paper, we show how to ensure the convexity of the fused lasso signal approximation problem…
Semidefinite programs (SDP) are important in learning and combinatorial optimization with numerous applications. In pursuit of low-rank solutions and low complexity algorithms, we consider the Burer--Monteiro factorization approach for solving SDPs. We show that all approximate local optima are global optima for the penalty formulation of appropriately rank-constrained SDPs as long as the number of constraints scales sub-quadratically with the desired rank of the optimal solution. Our result is based on a simple penalty function formulation of the rank-constrained SDP along with a smoothed ana…
