The numerical properties of algorithms for finding the intersection of sets depend to some extent on the regularity of the sets, but even more importantly on the regularity of the intersection. The alternating projection algorithm of von Neumann has been shown to converge locally at a linear rate dependent on the regularity modulus of the intersection. In many applications, however, the sets in question come from inexact measurements that are matched to idealized models. It is unlikely that any such problems in applications will enjoy metrically regular intersection, let alone set intersection. We explore a regularization strategy that generates an intersection with the desired regularity properties. The regularization, however, can lead to a significant increase in computational complexity. In a further refinement, we investigate and prove linear convergence of an approximate alternating projection algorithm. The analysis provides a regularization strategy that fits naturally with many ill-posed inverse problems, and a mathematically sound stopping criterion for extrapolated, approximate algorithms. The theory is demonstrated on the phase retrieval problem with experimental data. The conventional early termination applied in practice to unregularized, consistent problems in diffraction imaging can be justified fully in the framework of this analysis providing, for the first time, proof of convergence of alternating approximate projections for finite dimensional, consistent phase retrieval problems.
Sets of bilinear constraints are important in various machine learning models. Mathematically, they are hyperbolas in a product space. In this paper, we give a complete formula for projections onto sets of bilinear constraints or hyperbolas in a general Hilbert space.
We consider the set of pairs of orthogonal vectors in Hilbert space, which is also called the cross because it is the union of the horizontal and vertical axes in the Euclidean plane when the underlying space is the real line. Crosses, which are nonconvex sets, play a significant role in various branches of nonsmooth analysis such as feasibility problems and optimization problems. In this work, we study crosses and show that in infinite-dimensional settings, they are never weakly (sequentially) closed. Nonetheless, crosses do turn out to be proximinal (i.e., they always admit projections) and …
This article describes a set of methods for quickly computing the solution to the regularized optimal transport problem. It generalizes and improves upon the widely-used iterative Bregman projections algorithm (or Sinkhorn--Knopp algorithm). We first propose to rely on regularized nonlinear acceleration schemes. In practice, such approaches lead to fast algorithms, but their global convergence is not ensured. Hence, we next propose a new algorithm with convergence guarantees. The idea is to overrelax the Bregman projection operators, allowing for faster convergence. We propose a simple method …
This paper focuses on designing a unified approach for computing the projection onto the intersection of an $\ell_1$ ball/sphere and an $\ell_2$ ball/sphere. We show that the major computational efforts of solving these problems all rely on finding the root of the same piecewisely quadratic function, and then propose a unified numerical method to compute the root. In particular, we design breakpoint search methods with/without sorting incorporated with bisection, secant and Newton methods to find the interval containing the root, on which the root has a closed form. It can be shown that our pr…
We provide an elementary proof of a simple, efficient algorithm for computing the Euclidean projection of a point onto the probability simplex. We also show an application in Laplacian K-modes clustering.
The goal of this paper is to study approaches to bridge the gap between first-order and second-order type methods for composite convex programs. Our key observations are: i) Many well-known operator splitting methods, such as forward-backward splitting (FBS) and Douglas-Rachford splitting (DRS), actually define a fixed-point mapping; ii) The optimal solutions of the composite convex program and the solutions of a system of nonlinear equations derived from the fixed-point mapping are equivalent. Solving this kind of system of nonlinear equations enables us to develop second-order type methods. …
We study the projection onto the set of feasible inputs and the set of feasible solutions of a polynomial optimisation problem (POP). Our motivation is increasing the robustness of solvers for POP: Without a priori guarantees of feasibility of a particular instance, one should like to perform the projection onto the set of feasible inputs prior to running a solver. Without a certificate of optimality, one should like to project the output of the solver onto the set of feasible solutions subsequently. We study the computational complexity, formulations, and convexifications of the projections. …
It is well-known that the sequence of iterations of the composition of projections onto closed affine subspaces converges linearly to the projection onto the intersection of the affine subspaces when the sum of the corresponding linear subspaces is closed. Inspired by this, in this work, we systematically study the relation between the projection onto intersection of halfspaces and hyperplanes, and the composition of projections onto halfspaces and hyperplanes. In addition, as by-products, we provide the Karush-Kuhn-Tucker conditions for characterizing the optimal solution of convex optimizati…
In Hilbert space setting we prove local lipchitzness of projections onto parametric polyhedral sets represented as solutions to systems of inequalities and equations with parameters appearing both in left-hand-sides and right-hand-sides of the constraints. In deriving main results we assume that data are locally Lipschitz functions of parameter and the relaxed constant rank constraint qualification condition is satisfied.
This mini-paper presents a fast and simple algorithm to compute the projection onto the canonical simplex $\triangle^n$. Utilizing the Moreau's identity, we show that the problem is essentially a univariate minimization and the objective function is strictly convex and continuously differentiable. Moreover, it is shown that there are at most n candidates which can be computed explicitly, and the minimizer is the only one that falls into the correct interval.
We develop a practical approach to semidefinite programming (SDP) that includes the von Neumann entropy, or an appropriate variant, as a regularization term. In particular we solve the dual of the regularized program, demonstrating how a carefully chosen randomized trace estimator can be used to estimate dual gradients effectively. We also introduce specialized optimization approaches for common SDP, specifically SDP with diagonal constraint and the problem of the determining the spectral projector onto the span of extremal eigenvectors. We validate our approach on such problems with applicati…
We address the problem of projecting a point onto a quadratic hypersurface, more specifically a central quadric. We show how this problem reduces to finding a given root of a scalar-valued nonlinear function. We completely characterize one of the optimal solutions of the projection as either the unique root of this nonlinear function on a given interval, or as a point that belongs to a finite set of computable solutions. We then leverage this projection and the recent advancements in splitting methods to compute the projection onto the intersection of a box and a quadratic hypersurface with al…
A wedge (i.e., a closed nonempty set in the Euclidean space stable under addition and multiplication with non-negative scalars) induces by a standard way a semi-order (a reflexive and transitive binary relation) in the space. The wedges admitting isotone metric projection with respect to the semi-order induced by them are characterized. The obtained result is used to show that the monotone wedge (called monotone cone in regression theory) admits isotone projection.
The basic tool for solving problems in metric geometry and isotonic regression is the metric projection onto closed convex cones. Isotonicity of these projections with respect to a given order relation can facilitate finding the solutions of the above problems. In the recent note "A. B. Németh and S.Z. Németh: Isotonic regression and isotonic projection. Linear Algebra and its Applications, 494: 80-89 (2016)" this problem was studied for the coordinate-wise ordering. This study was the starting point for further investigations, such as the ones presented here. The order relation in the Euclide…
The Euclidean projection onto a convex set is an important problem that arises in numerous constrained optimization tasks. Unfortunately, in many cases, computing projections is computationally demanding. In this work, we focus on projection problems where the constraints are smooth and the number of constraints is significantly smaller than the dimension. The runtime of existing approaches to solving such problems is either cubic in the dimension or polynomial in the inverse of the target accuracy. Conversely, we propose a simple and efficient primal-dual approach, with a runtime that scales …
In this paper, we focus on the analysis of the regularized Wasserstein barycenter problem. We provide uniqueness and a characterization of the barycenter for two important classes of probability measures: (i) Gaussian distributions and (ii) $q$-Gaussian distributions; each regularized by a particular entropy functional. We propose an algorithm based on gradient projection method in the space of matrices in order to compute these regularized barycenters. We also consider a general class of $φ$-exponential measures, for which only the non-regularized barycenter is studied. Finally, we numericall…
A new deconvolution algorithm based on orthogonal projections onto the epigraph set of a convex cost function is presented. In this algorithm, the dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. As the utilized cost function is a convex function in $R^N$, the corresponding epigraph set is also a convex set in $R^{N+1}$. The deconvolution algorithm starts with an arbitrary initial estimate in $R^{N+1}$. At each step of the iterative algorithm, first deconvolution projections are performed onto the epigraphs, later an orthogonal pro…
In this paper, we provide a finitely terminated yet efficient approach to compute the Euclidean projection onto the ordered weighted $\ell_1$ (OWL1) norm ball. In particular, an efficient semismooth Newton method is proposed for solving the dual of a reformulation of the original projection problem. Global and local quadratic convergence results, as well as the finite termination property, of the algorithm are proved. Numerical comparisons with the two best-known methods demonstrate the efficiency of our method. In addition, we derive the generalized Jacobian of the studied projector which, we…
By using Moreau's decomposition theorem for projecting onto cones, the problem of projecting onto a simplicial cone is reduced to finding the unique solution of a nonsmooth system of equations. It is shown that Picard's method applied to the system of equations associated to the problem of projecting onto a simplicial cone generates a sequence that converges linearly to the solution of the system. Numerical experiments are presented making the comparison between Picard's and semi-smooth Newton's methods to solve the nonsmooth system associated with the problem of projecting a point onto a simp…
Both wavelet denoising and denosing methods using the concept of sparsity are based on soft-thresholding. In sparsity based denoising methods, it is assumed that the original signal is sparse in some transform domains such as the wavelet domain and the wavelet subsignals of the noisy signal are projected onto L1-balls to reduce noise. In this lecture note, it is shown that the size of the L1-ball or equivalently the soft threshold value can be determined using linear algebra. The key step is an orthogonal projection onto the epigraph set of the L1-norm cost function.
Approximate Projections onto the Positive Semidefinite Cone Using Randomization
School of Electrical and Electronic Engineering, The University of Sheffield（谢菲尔德大学电气与电子工程学院）；Department of Electrical Engineering, Columbia University（哥伦比亚大学电气工程系）
We show that the minimum distance projection in the L1-norm from an interior point onto the boundary of a convex set is achieved by a single, unidimensional projection. Application of this characterization when the convex set is a polyhedron leads to either an elementary minmax problem or a set of easily solved linear programs, depending upon whether the polyhedron is given as the intersection of a set of half spaces or as the convex hull of a set of extreme points. The outcome is an easier and more straightforward derivation of the special case results given in a recent paper by Briec.
This paper introduces the monotone extended second-order cone (MESOC), which is related to the monotone cone and the second-order cone. Some properties of the MESOC are presented and its dual cone is computed. Projecting onto the MESOC is reduced to the pool-adjacent-violators algorithm (PAVA) of isotonic regression. An application of MESOC to portfolio optimisation is provided. Some broad descriptions of possible MESOC-regression models are also outlined.
Two new optimization techniques based on projections onto convex space (POCS) framework for solving convex and some non-convex optimization problems are presented. The dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. If the cost function is a convex function in R^N the corresponding set is a convex set in R^(N+1). The iterative optimization approach starts with an arbitrary initial estimate in R^(N+1) and an orthogonal projection is performed onto one of the sets in a sequential manner at each step of the optimization problem. The …
In this paper, the problem of computing the projection, and therefore the minimum distance, from a point onto a Minkowski sum of general convex sets is studied. Our approach is based on the minimum norm duality theorem originally stated by Nirenberg and the Nesterov smoothing techniques. It is shown that projection points onto a Minkowski sum of sets can be represented as the sum of points on constituent sets so that, at these points, all of the sets share the same normal vector which is the negative of the dual solution. The proposed NESMINO algorithm improves the theoretical bound on number …
Projections onto sets are used in a wide variety of methods in optimization theory but not every method that uses projections really belongs to the class of projection methods as we mean it here. Here projection methods are iterative algorithms that use projections onto sets while relying on the general principle that when a family of (usually closed and convex) sets is present then projections (or approximate projections) onto the given individual sets are easier to perform than projections onto other sets (intersections, image sets under some transformation, etc.) that are derived from the g…
Two new optimization techniques based on projections onto convex space (POCS) framework for solving convex optimization problems are presented. The dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. If the cost function is a convex function in R^N the corresponding set is also a convex set in R^{N+1}. The iterative optimization approach starts with an arbitrary initial estimate in R^{N+1} and an orthogonal projection is performed onto one of the sets in a sequential manner at each step of the optimization problem. The method provides…
The metric projection onto an order nonnegative cone from the metric projection onto the corresponding order cone is derived. Particularly, we can use Pool Adjacent Violators-type algorithms developed for projecting onto the monotone cone for projecting onto the monotone nonnegative cone too.
Our study focuses on determining the best weight windows for a weighted moving average smoother under squared loss. We show that there exists an optimal weight window that is symmetrical around its center. We study the class of tapered weight windows, which decrease in weight as they move away from the center. We formulate the corresponding least squares problem as a quadratic program and finally as a projection of the origin onto a convex polytope. Additionally, we provide some analytical solutions to the best window when some conditions are met on the input data.
Mixed norms that promote structured sparsity have numerous applications in signal processing and machine learning problems. In this work, we present a new algorithm, based on a Newton root search technique, for computing the projection onto the $\ell_{\infty,1}$ ball, which has found application in cognitive neuroscience and classification tasks. Numerical simulations show that our proposed method is between 8 and 10 times faster on average, and up to 20 times faster for very sparse solutions, than the previous state of the art. Tests on real functional magnetic resonance image data show that,…
Joint sparsity offers powerful structural cues for feature selection, especially for variables that are expected to demonstrate a "grouped" behavior. Such behavior is commonly modeled via group-lasso, multitask lasso, and related methods where feature selection is effected via mixed-norms. Several mixed-norm based sparse models have received substantial attention, and for some cases efficient algorithms are also available. Surprisingly, several constrained sparse models seem to be lacking scalable algorithms. We address this deficiency by presenting batch and online (stochastic-gradient) optim…
Bilkent University（比尔肯特大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
A new signal processing framework based on making orthogonal Projections onto the Epigraph Set of a Convex cost function (PESC) is developed. In this way it is possible to solve convex optimization problems using the well-known Projections onto Convex Set (POCS) approach. In this algorithm, the dimension of the minimization problem is lifted by one and a convex set corresponding to the epigraph of the cost function is defined. If the cost function is a convex function in $R^N$, the corresponding epigraph set is also a convex set in R^{N+1}. The PESC method provides globally optimal solutions f…
The von Neumann-Halperin method of alternating projections converges strongly to the projection of a given point onto the intersection of finitely many closed affine subspaces. We propose acceleration schemes making use of two ideas: Firstly, each projection onto an affine subspace identifies a hyperplane of codimension 1 containing the intersection, and secondly, it is easy to project onto a finite intersection of such hyperplanes. We give conditions for which our accelerations converge strongly. Finally, we perform numerical experiments to show that these accelerations perform well for a mat…
The space of finite games can be decomposed into three orthogonal subspaces [5], which are the subspaces of pure potential games, nonstrategic games and pure harmonic games. The orthogonal projections onto these subspaces are represented as the Moore-Penrose inverses of the corresponding linear operators (i.e., matrices) [5]. Although the representation is compact and nice, no analytic method is given to calculate Moore- Penrose inverses of these linear operators. Hence using their results, one cannot verify whether a finite game belongs to one of these subspaces. In this paper, jumping over c…
We propose algorithms and software for computing projections onto the intersection of multiple convex and non-convex constraint sets. The software package, called SetIntersectionProjection, is intended for the regularization of inverse problems in physical parameter estimation and image processing. The primary design criterion is working with multiple sets, which allows us to solve inverse problems with multiple pieces of prior knowledge. Our algorithms outperform the well known Dykstra's algorithm when individual sets are not easy to project onto because we exploit similarities between constr…
This paper is devoted to the general problem of projection onto a polyhedral convex cone generated by a finite set of generators.This problem is reformulated into projection onto the polytope obtained by simple truncation of the original cone. Then it can be solved with just two closely related projections onto the same bounded polytope. This approach's computational performance is conditioned by the crucial tool's efficiency for solving the fundamental problem of finding the least norm element in a convex hull of a given finite set of points. In our numerical experiments, we used for this pur…
We provide formulas for projectors onto a polyhedral set, i.e. the intersection of a finite number of halfspaces. To this aim we formulate the problem of finding the projection as a convex optimization problem and we solve explicitly sufficient and necessary optimality conditions. This approach has already been successfully applied in deriving formulas for projection onto the intersection of two halfspaces. We also discuss possible generalizations to Banach spaces.
Extending Meshulam's result on the boundedness of orbits of relaxed projections onto affine subspaces from finite to infinite-dimensional Hilbert spaces
