In applications throughout science and engineering one is often faced with the challenge of solving an ill-posed inverse problem, where the number of available measurements is smaller than the dimension of the model to be estimated. However in many practical situations of interest, models are constrained structurally so that they only have a few degrees of freedom relative to their ambient dimension. This paper provides a general framework to convert notions of simplicity into convex penalty functions, resulting in convex optimization solutions to linear, underdetermined inverse problems. The class of simple models considered are those formed as the sum of a few atoms from some (possibly infinite) elementary atomic set; examples include well-studied cases such as sparse vectors and low-rank matrices, as well as several others including sums of a few permutations matrices, low-rank tensors, orthogonal matrices, and atomic measures. The convex programming formulation is based on minimizing the norm induced by the convex hull of the atomic set; this norm is referred to as the atomic norm. The facial structure of the atomic norm ball carries a number of favorable properties that are useful for recovering simple models, and an analysis of the underlying convex geometry provides sharp estimates of the number of generic measurements required for exact and robust recovery of models from partial information. These estimates are based on computing the Gaussian widths of tangent cones to the atomic norm ball. When the atomic set has algebraic structure the resulting optimization problems can be solved or approximated via semidefinite programming. The quality of these approximations affects the number of measurements required for recovery. Thus this work extends the catalog of simple models that can be recovered from limited linear information via tractable convex programming.
We study multivariate normal models that are described by linear constraints on the inverse of the covariance matrix. Maximum likelihood estimation for such models leads to the problem of maximizing the determinant function over a spectrahedron, and to the problem of characterizing the image of the positive definite cone under an arbitrary linear projection. These problems at the interface of statistics and optimization are here examined from the perspective of convex algebraic geometry.
We propose algorithms and software for computing projections onto the intersection of multiple convex and non-convex constraint sets. The software package, called SetIntersectionProjection, is intended for the regularization of inverse problems in physical parameter estimation and image processing. The primary design criterion is working with multiple sets, which allows us to solve inverse problems with multiple pieces of prior knowledge. Our algorithms outperform the well known Dykstra's algorithm when individual sets are not easy to project onto because we exploit similarities between constr…
We present a new inverse optimization methodology for multi-objective convex optimization that accommodates an input solution that may not be Pareto optimal and determines a weight vector that produces a Pareto optimal solution that approximates the input solution and preserves the decision maker's intention encoded in it. We introduce a notion of trade-off preservation, which we use as a measure of similarity for approximating the input solution, and show its connection with minimizing an optimality gap. Our inverse model maintains the complexity of the traditional inverse convex models. We p…
This work is concerned with linear inverse problems where a distributed parameter is known a priori to only take on values from a given discrete set. This property can be promoted in Tikhonov regularization with the aid of a suitable convex but nondifferentiable regularization term. This allows applying standard approaches to show well-posedness and convergence rates in Bregman distance. Using the specific properties of the regularization term, it can be shown that convergence (albeit without rates) actually holds pointwise. Furthermore, the resulting Tikhonov functional can be minimized effic…
In this article we dwell into the class of so called ill posed Linear Inverse Problems (LIP) in machine learning, which has become almost a classic in recent times. The fundamental task in an LIP is to recover the entire signal / data from its relatively few random linear measurements. Such problems arise in variety of settings with applications ranging from medical image processing, recommender systems etc. We provide an exposition to the convex duality of the linear inverse problems, and obtain a novel and equivalent convex-concave min-max reformulation that gives rise to simple ascend-desce…
The theory of convex risk functions has now been well established as the basis for identifying the families of risk functions that should be used in risk averse optimization problems. Despite its theoretical appeal, the implementation of a convex risk function remains difficult, as there is little guidance regarding how a convex risk function should be chosen so that it also well represents one's own risk preferences. In this paper, we address this issue through the lens of inverse optimization. Specifically, given solution data from some (forward) risk-averse optimization problems we develop …
We describe two nonconventional algorithms for linear regression, called GAME and CLASH. The salient characteristics of these approaches is that they exploit the convex $\ell_1$-ball and non-convex $\ell_0$-sparsity constraints jointly in sparse recovery. To establish the theoretical approximation guarantees of GAME and CLASH, we cover an interesting range of topics from game theory, convex and combinatorial optimization. We illustrate that these approaches lead to improved theoretical guarantees and empirical performance beyond convex and non-convex solvers alone.
Several applications in medical imaging and non-destructive material testing lead to inverse elliptic coefficient problems, where an unknown coefficient function in an elliptic PDE is to be determined from partial knowledge of its solutions. This is usually a highly non-linear ill-posed inverse problem, for which unique reconstructability results, stability estimates and global convergence of numerical methods are very hard to achieve. The aim of this note is to point out a new connection between inverse coefficient problems and semidefinite programming that may help addressing these challenge…
This note conducts a comparative study of some approximating properties of the metric projection, generalized projection, and generalized metric projection in uniformly convex and uniformly smooth Banach spaces. We prove that the inverse images of the metric projections are closed and convex cones, but they are not necessarily convex. In contrast, inverse images of the generalized projection are closed and convex cones. Furthermore, the inverse images of the generalized metric projection are neither a convex set nor a cone. We also prove that the distance from a point to its projection at a co…
A Proximal Decomposition Method for Solving Convex Variational Inverse Problems
UPMC Université Paris 06（巴黎第六大学）；Laboratoire Jacques-Louis Lions – UMR 7598（雅克-路易·利翁斯实验室）；Université Paris-Est（巴黎东大学）；Institut Gaspard Monge and UMR CNRS 8049（加斯帕尔·蒙日研究所）
We consider a Cournot oligopoly model where multiple suppliers (oligopolists) compete by choosing quantities. We compare the social welfare achieved at a Cournot equilibrium to the maximum possible, for the case where the inverse market demand function is convex. We establish a lower bound on the efficiency of Cournot equilibria in terms of a scalar parameter derived from the inverse demand function, namely, the ratio of the slope of the inverse demand function at the Cournot equilibrium to the average slope of the inverse demand function between the Cournot equilibrium and a social optimum. A…
The affine inverse eigenvalue problem consists of identifying a real symmetric matrix with a prescribed set of eigenvalues in an affine space. Due to its ubiquity in applications, various instances of the problem have been widely studied in the literature. Previous algorithmic solutions were typically nonconvex heuristics and were often developed in a case-by-case manner for specific structured affine spaces. In this short note we describe a general family of convex relaxations for the problem by reformulating it as a question of checking feasibility of a system of polynomial equations, and th…
The set of controllers stabilizing a linear system is generally non-convex in the parameter space. In the case of two-parameter controller design (e.g. PI control or static output feedback with one input and two outputs), we observe however that quite often for benchmark problem instances, the set of stabilizing controllers seems to be convex. In this note we use elementary techniques from real algebraic geometry (resultants and Bezoutian matrices) to explain this phenomenon. As a byproduct, we derive a convex linear matrix inequality (LMI) formulation of two-parameter fixed-order controller d…
We study non-convex subgradient flows for training two-layer ReLU neural networks from a convex geometry and duality perspective. We characterize the implicit bias of unregularized non-convex gradient flow as convex regularization of an equivalent convex model. We then show that the limit points of non-convex subgradient flows can be identified via primal-dual correspondence in this convex optimization problem. Moreover, we derive a sufficient condition on the dual variables which ensures that the stationary points of the non-convex objective are the KKT points of the convex objective, thus pr…
University of Florida（佛罗里达大学）；Georgia State University（佐治亚州立大学）
Optimization algorithms for solving nonconvex inverse problem have attracted significant interests recently. However, existing methods require the nonconvex regularization to be smooth or simple to ensure convergence. In this paper, we propose a novel gradient descent type algorithm, by leveraging the idea of residual learning and Nesterov's smoothing technique, to solve inverse problems consisting of general nonconvex and nonsmooth regularization with provable convergence. Moreover, we develop a neural network architecture intimating this algorithm to learn the nonlinear sparsity transformati…
We study the convex geometry of the forward reach sets for integrator dynamics in finite dimensions with bounded control. We derive closed-form expressions for the volume and the diameter (i.e., maximal width) of these sets in terms of the state space dimension, control bound, and time. These results are novel, and use convex analysis to give an analytical handle on the "size" of the integrator reach set. Several concrete examples are provided to illustrate our results. We envision that the ideas presented here will motivate further theoretical and algorithmic development in reach set computat…
We introduce a concept that generalizes several different notions of a "centerpoint" in the literature. We develop an oracle-based algorithm for convex mixed-integer optimization based on centerpoints. Further, we show that algorithms based on centerpoints are "best possible" in a certain sense. Motivated by this, we establish several structural results about this concept and provide efficient algorithms for computing these points. Our main motivation is to understand the complexity of oracle based convex mixed-integer optimization.
Convex algebraic geometry concerns the interplay between optimization theory and real algebraic geometry. Its objects of study include convex semialgebraic sets that arise in semidefinite programming and from sums of squares. This article compares three notions of duality that are relevant in these contexts: duality of convex bodies, duality of projective varieties, and the Karush-Kuhn-Tucker conditions derived from Lagrange duality. We show that the optimal value of a polynomial program is an algebraic function whose minimal polynomial is expressed by the hypersurface projectively dual to the…
The non-convex quadratic orogramming problem and the non-monotone linear complementarity problem are NP-complete problems. In this paper we first show taht the inverse problem of determinning a KKT point of the non-convex quadratic programming problem is polynomial. We then show that the inverse problems of non-monotone linear complementarity problem are polynomial solvable in some cases, and in another case is NP-hard. Therefore we solve an open question raised by Heuberger on inverse NP-hard problems and prove that CoNP=NP.
A sharp phase transition emerges in convex programs when solving the linear inverse problem, which aims to recover a structured signal from its linear measurements. This paper studies this phenomenon in theory under Gaussian random measurements. Different from previous studies, in this paper, we consider convex programs with multiple prior constraints. These programs are encountered in many cases, for example, when the signal is sparse and its $\ell_2$ norm is known beforehand, or when the signal is sparse and non-negative simultaneously. Given such a convex program, to analyze its phase trans…
We consider several basic questions pertaining to the geometry of image of a general quadratic map. In general the image of a quadratic map is non-convex, although there are several known classes of quadratic maps when the image is convex. Remarkably, even when the image is not convex it often exhibits hidden convexity: a surprising efficiency of convex relaxation to address various geometric questions by reformulating them in terms of convex optimization problems. In this paper we employ this strategy and put forward several algorithms that solve the following problems pertaining to the image…
The geometry conjecture, which was posed nearly a quarter of a century ago, states that the fixed point set of the composition of projectors onto nonempty closed convex sets in Hilbert space is actually equal to the intersection of certain translations of the underlying sets. In this paper, we provide a complete resolution of the geometry conjecture. Our proof relies on monotone operator theory. We revisit previously known results and provide various illustrative examples. Comments on the numerical computation of the quantities involved are also presented.
Inverse scattering problems of the reconstructions of physical properties of a medium from boundary measurements are substantially challenging ones. This work aims to verify the performance on experimental data of a newly developed convexification method for a 3D coefficient inverse problem for the case of objects buried in a sandbox a fixed frequency and the point source moving along an interval of a straight line. Using a special Fourier basis, the method of this work strongly relies on a new derivation of a boundary value problem for a system of coupled quasilinear elliptic equations. This …
Inverse optimization (IO) aims to determine optimization model parameters from observed decisions. However, IO is not part of a data scientist's toolkit in practice, especially as many general-purpose machine learning packages are widely available as an alternative. When encountering IO, practitioners face the question of when, or even whether, investing in developing IO methods is worthwhile. Our paper provides a starting point toward answering these questions, focusing on the problem of imputing the objective function of a parametric convex optimization problem. We compare the predictive per…
Convex geometries form a subclass of closure systems with unique criticals, or $UC$-systems. We show that the $F$-basis introduced in [1] for $UC$-systems, becomes optimum in convex geometries, in two essential parts of the basis: right sides (conclusions) of binary implications and left sides (premises) of non-binary ones. The right sides of non-binary implications can also be optimized, when the convex geometry either satisfies the Carousel property, or does not have $D$-cycles. The latter generalizes a result of P.L.~Hammer and A.~Kogan for acyclic Horn Boolean functions. Convex geometries …
A new convenient method of describing flat convex compact sets is proposed. It generalizes classical trigonometric functions $\sin$ and $\cos$. Apparently, this method may be very useful for explicit description of solutions of optimal control problems with two-dimensional control. Using this method a series of sub-Finsler problems with two-dimensional control lying in an arbitrary convex set $Ω$ is investigated. Namely, problems on the Heisenberg, Engel, and Cartan groups and also Grushin's and Martinet's cases are considered. A particular attention is paid to the case when $Ω$ is a polygon.
We review several (and provide new) results on the theory of moments, sums of squares and basic semi-algebraic sets when convexity is present. In particular, we show that under convexity, the hierarchy of semidefinite relaxations for polynomial optimization simplifies and has finite convergence, a highly desirable feature as convex problems are in principle easier to solve. In addition, if a basic semi-algebraic set K is convex but its defining polynomials are not, we provide a certificate of convexity if a sufficient (and almost necessary) condition is satified. This condition can be checked …
We focus on nonconvex and nonsmooth minimization problems with a composite objective, where the differentiable part of the objective is freed from the usual and restrictive global Lipschitz gradient continuity assumption. This longstanding smoothness restriction is pervasive in first order methods (FOM), and was recently circumvent for convex composite optimization by Bauschke, Bolte and Teboulle, through a simple and elegant framework which captures, all at once, the geometry of the function and of the feasible set. Building on this work, we tackle genuine nonconvex problems. We first complem…
The inverse linear-quadratic optimal control problem is a system identification problem whose aim is to recover the quadratic cost function and hence the closed-loop system matrices based on observations of optimal trajectories. In this paper, the discrete-time, finite-horizon case is considered, where the agents are also assumed to be homogeneous and indistinguishable. The latter means that the agents all have the same dynamics and objective functions and the observations are in terms of "snap shots" of all agents at different time instants, but what is not known is "which agent moved where" …
For the first time, we develop in this paper the globally convergent convexification numerical method for a Coefficient Inverse Problem for the 3D Helmholtz equation for the case when the backscattering data are generated by a point source running along an interval of a straight line and the wavenumber is fixed. Thus, by varying the wavenumber, one can reconstruct the dielectric constant depending not only on spatial variables but the wavenumber (i.e. frequency) as well. Our approach relies on a new derivation of a boundary value problem for a system of coupled quasilinear elliptic partial dif…
