We consider real polynomials in finitely many variables. Let the variables consist of finitely many blocks that are allowed to overlap in a certain way. Let the solution set of a finite system of polynomial inequalities be given where each inequality involves only variables of one block. We investigate polynomials that are positive on such a set and sparse in the sense that each monomial involves only variables of one block. In particular, we derive a short and direct proof for Lasserre's theorem of the existence of sums of squares certificates respecting the block structure. The motivation for the results can be found in the literature and stems from numerical methods using semidefinite programming to simulate or control discrete-time behaviour of systems.
Motivated by the minimax concave penalty based variable selection in high-dimensional linear regression, we introduce a simple scheme to construct structured semiconvex sparsity promoting functions from convex sparsity promoting functions and their Moreau envelopes. Properties of these functions are developed by leveraging their structure. In particular, we provide sparsity guarantees for the general family of functions. We further study the behavior of the proximity operators of several special functions including indicator functions of closed convex sets, piecewise quadratic functions, and t…
Compressive sensing (CS) exploits sparsity to recover sparse or compressible signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity is also used to enhance interpretability in machine learning and statistics applications: While the ambient dimension is vast in modern data analysis problems, the relevant information therein typically resides in a much lower dimensional space. However, many solutions proposed nowadays do not leverage the true underlying structure. Recent results in CS extend the simple sparsity idea to more sophisticated {\em structured} sparsity models,…
The parameters of a neural network are naturally organized in groups, some of which might not contribute to its overall performance. To prune out unimportant groups of parameters, we can include some non-differentiable penalty to the objective function, and minimize it using proximal gradient methods. In this paper, we derive the weighted proximal operator, which is a necessary component of these proximal methods, of two structured sparsity inducing penalties. Moreover, they can be approximated efficiently with a numerical solver, and despite this approximation, we prove that existing converge…
Fundamental in matrix algebra and its applications, a \emph{generalized inverse} of a real matrix $A$ is a matrix $H$ that satisfies the Moore-Penrose (M-P) property $AHA=A$. If $H$ also satisfies the additional useful M-P property, $HAH=H$, it is called a \emph{reflexive generalized inverse}. Reflexivity is equivalent to minimum rank, so we are particularly interested in reflexive generalized inverses. We consider aspects of symmetry related to the calculation of a \emph{sparse} reflexive generalized inverse of $A$. As is common, and following Lee and Fampa (2018) for calculating sparse gener…
Penalty functions or regularization terms that promote structured solutions to optimization problems are of great interest in many fields. Proposed in this work is a nonconvex structured sparsity penalty that promotes one-sparsity within arbitrary overlapping groups in a vector. This allows one to enforce mutual exclusivity between components within solutions to optimization problems. We show multiple example use cases (including a total variation variant), demonstrate synergy between it and other regularizers, and propose an algorithm to efficiently solve problems regularized or constrained b…
We consider a class of sparse learning problems in high dimensional feature space regularized by a structured sparsity-inducing norm which incorporates prior knowledge of the group structure of the features. Such problems often pose a considerable challenge to optimization algorithms due to the non-smoothness and non-separability of the regularization term. In this paper, we focus on two commonly adopted sparsity-inducing regularization terms, the overlapping Group Lasso penalty $l_1/l_2$-norm and the $l_1/l_\infty$-norm. We propose a unified framework based on the augmented Lagrangian method,…
Georgia Institute of Technology（佐治亚理工学院）；Mitsubishi Electric Research Labs（三菱电机研究实验室）；Carnegie Mellon University（卡内基梅隆大学）
Several convex formulation methods have been proposed previously for statistical estimation with structured sparsity as the prior. These methods often require a carefully tuned regularization parameter, often a cumbersome or heuristic exercise. Furthermore, the estimate that these methods produce might not belong to the desired sparsity model, albeit accurately approximating the true parameter. Therefore, greedy-type algorithms could often be more desirable in estimating structured-sparse parameters. So far, these greedy methods have mostly focused on linear statistical models. In this paper w…
University of California（加利福尼亚大学）；INRIA（法国国家信息与自动化研究所）；Laboratoire d’Informatique de l’Ecole Normale Supérieure（巴黎高等师范学院计算机实验室）；CNRS（法国国家科学研究中心）
We consider a class of learning problems regularized by a structured sparsity-inducing norm defined as the sum of l_2- or l_infinity-norms over groups of variables. Whereas much effort has been put in developing fast optimization techniques when the groups are disjoint or embedded in a hierarchy, we address here the case of general overlapping groups. To this end, we present two different strategies: On the one hand, we show that the proximal operator associated with a sum of l_infinity-norms can be computed exactly in polynomial time by solving a quadratic min-cost flow problem, allowing the …
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；Université de Toulouse（图卢兹大学）；CNRS（法国国家科学研究中心）；UPS IMT（图卢兹第三大学数学研究所）；Institut de Recherche Technologique Saint-Exupéry（圣埃克苏佩里技术研究所）
We consider a deep structured linear network under sparsity constraints. We study sharp conditions guaranteeing the stability of the optimal parameters defining the network. More precisely, we provide sharp conditions on the network architecture and the sample under which the error on the parameters defining the network scales linearly with the reconstruction error (i.e. the risk). Therefore, under these conditions, the weights obtained with a successful algorithms are well defined and only depend on the architecture of the network and the sample. The features in the latent spaces are stably d…
Generalized inverses are important in statistics and other areas of applied matrix algebra. A \emph{generalized inverse} of a real matrix $A$ is a matrix $H$ that satisfies the Moore-Penrose (M-P) property $AHA=A$. If $H$ also satisfies the M-P property $HAH=H$, then it is called \emph{reflexive}. Reflexivity of a generalized inverse is equivalent to minimum rank, a highly desirable property. We consider aspects of symmetry related to the calculation of various \emph{sparse} reflexive generalized inverses of $A$. As is common, we use (vector) 1-norm minimization for both inducing sparsity and …
Partial calmness is a celebrated but restrictive property of bilevel optimization problems whose presence opens a way to the derivation of Karush--Kuhn--Tucker-type necessary optimality conditions in order to characterize local minimizers. In the past, sufficient conditions for the validity of partial calmness have been investigated. In this regard, the presence of a linearly structured lower level problem has turned out to be beneficial. However, the associated literature suffers from inaccurate results. In this note, we clarify some regarding erroneous statements and visualize the underlying…
We provide a sparse version of the bounded degree SOS hierarchy BSOS [7] for polynomial optimization problems. It permits to treat large scale problems which satisfy a structured sparsity pattern. When the sparsity pattern satisfies the running intersection property this Sparse-BSOS hierarchy of semidefinite programs (with semidefinite constraints of fixed size) converges to the global optimum of the original problem. Moreover, for the class of SOS-convex problems, finite convergence takes place at the first step of the hierarchy, just as in the dense version.
INRIA（法国国家信息与自动化研究所）；Laboratoire d’Informatique de l’Ecole Normale Supérieure（巴黎高等师范学院计算机实验室）
Sparse methods for supervised learning aim at finding good linear predictors from as few variables as possible, i.e., with small cardinality of their supports. This combinatorial selection problem is often turned into a convex optimization problem by replacing the cardinality function by its convex envelope (tightest convex lower bound), in this case the L1-norm. In this paper, we investigate more general set-functions than the cardinality, that may incorporate prior knowledge or structural constraints which are common in many applications: namely, we show that for nondecreasing submodular set…
Learning the hub graphical Lasso model with the structured sparsity via an efficient algorithm
School of Mathematics, Southwest Jiaotong University（西南交通大学数学学院）；School of Computer and Computing Science, Zhejiang University City College（浙江大学城市学院计算机与计算科学学院）；Engineering Systems and Design, Singapore University of Technology and Design（新加坡科技设计大学工程系统与设计）
