The task of projecting onto $\ell_p$ norm balls is ubiquitous in statistics and machine learning, yet the availability of actionable algorithms for doing so is largely limited to the special cases of $p = \left\{ 0, 1,2, \infty \right\}$. In this paper, we introduce novel, scalable methods for projecting onto the $\ell_p$ ball for general $p>0$. For $p \geq1 $, we solve the univariate Lagrangian dual via a dual Newton method. We then carefully design a bisection approach for $p<1$, presenting theoretical and empirical evidence of zero or a small duality gap in the non-convex case. The success of our contributions is thoroughly assessed empirically, and applied to large-scale regularized multi-task learning and compressed sensing.
In this paper, we provide a finitely terminated yet efficient approach to compute the Euclidean projection onto the ordered weighted $\ell_1$ (OWL1) norm ball. In particular, an efficient semismooth Newton method is proposed for solving the dual of a reformulation of the original projection problem. Global and local quadratic convergence results, as well as the finite termination property, of the algorithm are proved. Numerical comparisons with the two best-known methods demonstrate the efficiency of our method. In addition, we derive the generalized Jacobian of the studied projector which, we…
In the present paper, the following convexity principle is proved: any closed convex multifunction, which is metrically regular in a certain uniform sense near a given point, carries small balls centered at that point to convex sets, even if it is perturbed by adding C^{1,1} smooth mappings with controlled Lipschizian behaviour. This result, which is valid for mappings defined on a subclass of uniformly convex Banach spaces, can be regarded as a set-valued generalization of the Polyak convexity principle. The latter, indeed, can be derived as a special case of the former. Such an extension of …
We study convergence of the trajectories of the Heavy Ball dynamical system, with constant damping coefficient, in the framework of convex and non-convex smooth optimization. By using the Polyak-Łojasiewicz condition, we derive new linear convergence rates for the associated trajectory, in terms of objective function values, without assuming uniqueness of the minimizer.
This paper establishes global convergence and provides global bounds of the convergence rate of the Heavy-ball method for convex optimization problems. When the objective function has Lipschitz-continuous gradient, we show that the Cesaro average of the iterates converges to the optimum at a rate of $O(1/k)$ where k is the number of iterations. When the objective function is also strongly convex, we prove that the Heavy-ball iterates converge linearly to the unique optimum.
Whereas the norm of a vector measures amplitude (and is a 1-homogeneous function), sparsity is measured by the 0-homogeneous l0 pseudonorm, which counts the number of nonzero components. We propose a family of conjugacies suitable for the analysis of 0-homogeneous functions. These conjugacies are derived from couplings between vectors, given by their scalar product divided by a 1-homogeneous normalizing factor. With this, we characterize the best convex lower approximation of a 0-homogeneous function on the unit ''ball'' of a normalization function (i.e. a norm without the requirement of subad…
Mixed norms that promote structured sparsity have numerous applications in signal processing and machine learning problems. In this work, we present a new algorithm, based on a Newton root search technique, for computing the projection onto the $\ell_{\infty,1}$ ball, which has found application in cognitive neuroscience and classification tasks. Numerical simulations show that our proposed method is between 8 and 10 times faster on average, and up to 20 times faster for very sparse solutions, than the previous state of the art. Tests on real functional magnetic resonance image data show that,…
This note conducts a comparative study of some approximating properties of the metric projection, generalized projection, and generalized metric projection in uniformly convex and uniformly smooth Banach spaces. We prove that the inverse images of the metric projections are closed and convex cones, but they are not necessarily convex. In contrast, inverse images of the generalized projection are closed and convex cones. Furthermore, the inverse images of the generalized metric projection are neither a convex set nor a cone. We also prove that the distance from a point to its projection at a co…
In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Nesterov later developed a globally accelerated version, Polyak's original algorithm remains simpler and more widely used in applications such as deep learning. Despite this popularity, the question of whether Heavy-ball is also globally accelerated or not has not been fully answered yet, and no convincing counterexample has been provided. This is largely due to the difficulty in finding an effective Lyapunov function: in…
The smallest enclosing circle problem asks for the circle of smallest radius enclosing a given set of finite points on the plane. This problem was introduced in the 19th century by Sylvester [17]. After more than a century, the problem remains very active. This paper is the continuation of our effort in shedding new light to classical geometry problems using advanced tools of convex analysis and optimization. We propose and study the following generalized version of the smallest enclosing circle problem: given a finite number of nonempty closed convex sets in a reflexive Banach space, find a b…
In this paper we consider a problem, called convex projection, of projecting a convex set onto a subspace. We will show that to a convex projection one can assign a particular multi-objective convex optimization problem, such that the solution to that problem also solves the convex projection (and vice versa), which is analogous to the result in the polyhedral convex case considered in arXiv:1507.00228. In practice, however, one can only compute approximate solutions in the (bounded or self-bounded) convex case, which solve the problem up to a given error tolerance. We will show that for appro…
A remarkable example of a nonempty closed convex set in the Euclidean plane for which the directional derivative of the metric projection mapping fails to exist was constructed by A. Shapiro. In this paper, we revisit and modify that construction to obtain a convex set with smooth boundary which possesses the same property.
In this work we consider an iterative method for solving the quasi-convex feasibility problem. We firstly introduce the so-called star subgradient projection operator and present some useful properties. We subsequently obtain a convergence result of the considered method by using properties of the introduced nonlinear operator.
We propose a generalization of the method of cyclic projections, which uses the lengths of projection steps carried out in the past to learn about the geometry of the problem and decides on this basis which projections to carry out in the future. We prove the convergence of this algorithm and illustrate its behavior in a first numerical study.
Momentum first-order optimization methods are the workhorses in various optimization tasks, e.g., in the training of deep neural networks. Recently, Lucas et al. (2019) proposed a method called Aggregated Heavy-Ball (AggHB) that uses multiple momentum vectors corresponding to different momentum parameters and averages these vectors to compute the update direction at each iteration. Lucas et al. (2019) show that AggHB is more stable than the classical Heavy-Ball method even with large momentum parameters and performs well in practice. However, the method was analyzed only for quadratic objectiv…
Both wavelet denoising and denosing methods using the concept of sparsity are based on soft-thresholding. In sparsity based denoising methods, it is assumed that the original signal is sparse in some transform domains such as the wavelet domain and the wavelet subsignals of the noisy signal are projected onto L1-balls to reduce noise. In this lecture note, it is shown that the size of the L1-ball or equivalently the soft threshold value can be determined using linear algebra. The key step is an orthogonal projection onto the epigraph set of the L1-norm cost function.
Finding a point in the intersection of a collection of closed convex sets, that is the convex feasibility problem, represents the main modeling strategy for many computational problems. In this paper we analyze new stochastic reformulations of the convex feasibility problem in order to facilitate the development of new algorithmic schemes. We also analyze the conditioning problem parameters using certain (linear) regularity assumptions on the individual convex sets. Then, we introduce a general random projection algorithmic framework, which extends to the random settings many existing projecti…
In this paper, we studied the equilibrium problem where the bi-function may be quasiconvex with respect to the second variable and the feasible set is the intersection of a finite number of convex sets. We propose a projection-algorithm, where the projection can be computed independently onto each component set. The convergence of the algorithm is investigated and numerical examples for a variational inequality problem involving affine fractional operator are provided to demonstrate the behavior of the algorithm.
This paper deals with the convex feasibility problem, where the feasible set is given as the intersection of a (possibly infinite) number of closed convex sets. We assume that each set is specified algebraically as a convex inequality, where the associated convex function is general (possibly non-differentiable). For finding a point satisfying all the convex inequalities we design and analyze random projection algorithms using special subgradient iterations and extrapolated stepsizes. Moreover, the iterate updates are performed based on parallel random observations of several constraint compon…
This paper is devoted to the general problem of projection onto a polyhedral convex cone generated by a finite set of generators.This problem is reformulated into projection onto the polytope obtained by simple truncation of the original cone. Then it can be solved with just two closely related projections onto the same bounded polytope. This approach's computational performance is conditioned by the crucial tool's efficiency for solving the fundamental problem of finding the least norm element in a convex hull of a given finite set of points. In our numerical experiments, we used for this pur…
We show that the minimum distance projection in the L1-norm from an interior point onto the boundary of a convex set is achieved by a single, unidimensional projection. Application of this characterization when the convex set is a polyhedron leads to either an elementary minmax problem or a set of easily solved linear programs, depending upon whether the polyhedron is given as the intersection of a set of half spaces or as the convex hull of a set of extreme points. The outcome is an easier and more straightforward derivation of the special case results given in a recent paper by Briec.
In this paper, we propose a new inexact version of the projected subgradient method to solve nondifferentiable constrained convex optimization problems. The method combine $ε$-subgradient method with a procedure to obtain a feasible inexact projection onto the constraint set. Asymptotic convergence results and iteration-complexity bounds for the sequence generated by the method employing the well known exogenous stepsizes, Polyak's stepsizes, and dynamic stepsizes are established.
We develop a distributed algorithm for convex Empirical Risk Minimization, the problem of minimizing large but finite sum of convex functions over networks. The proposed algorithm is derived from directly discretizing the second-order heavy-ball differential equation and results in an accelerated convergence rate, i.e, faster than distributed gradient descent-based methods for strongly convex objectives that may not be smooth. Notably, we achieve acceleration without resorting to the well-known Nesterov's momentum approach. We provide numerical experiments and contrast the proposed method with…
Two new optimization techniques based on projections onto convex space (POCS) framework for solving convex and some non-convex optimization problems are presented. The dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. If the cost function is a convex function in R^N the corresponding set is a convex set in R^(N+1). The iterative optimization approach starts with an arbitrary initial estimate in R^(N+1) and an orthogonal projection is performed onto one of the sets in a sequential manner at each step of the optimization problem. The …
This paper considers a conceptual version of a convex optimization algorithm whic is based on replacing a convex optimization problem with the root-finding problem for the approximate sub-differential mapping which is solved by repeated projection onto the epigraph of conjugate function. Whilst the projection problem is not exactly solvable in finite space-time it can be approximately solved up to arbitrary precision by simple iterative methods, which use linear support functions of the epigraph. It seems therefore useful to study computational characteristics of the idealized version of this …
The basic tool for solving problems in metric geometry and isotonic regression is the metric projection onto closed convex cones. Isotonicity of these projections with respect to a given order relation can facilitate finding the solutions of the above problems. In the recent note "A. B. Németh and S.Z. Németh: Isotonic regression and isotonic projection. Linear Algebra and its Applications, 494: 80-89 (2016)" this problem was studied for the coordinate-wise ordering. This study was the starting point for further investigations, such as the ones presented here. The order relation in the Euclide…
The Euclidean projection onto a convex set is an important problem that arises in numerous constrained optimization tasks. Unfortunately, in many cases, computing projections is computationally demanding. In this work, we focus on projection problems where the constraints are smooth and the number of constraints is significantly smaller than the dimension. The runtime of existing approaches to solving such problems is either cubic in the dimension or polynomial in the inverse of the target accuracy. Conversely, we propose a simple and efficient primal-dual approach, with a runtime that scales …
A local convergence result for abstract descent methods is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward--backward splitting method: iPiano---a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equival…
In this paper we present a new algorithmic realization of a projection-based scheme for general convex constrained optimization problem. The general idea is to transform the original optimization problem to a sequence of feasibility problems by iteratively constraining the objective function from above until the feasibility problem is inconsistent. For each of the feasibility problems one may apply any of the existing projection methods for solving it. In particular, the scheme allows the use of subgradient projections and does not require exact projections onto the constraints sets as in exis…
In this paper, we consider convex stochastic optimization problems arising in machine learning applications (e.g., risk minimization) and mathematical statistics (e.g., maximum likelihood estimation). There are two main approaches to solve such kinds of problems, namely the Stochastic Approximation approach (online approach) and the Sample Average Approximation approach, also known as the Monte Carlo approach, (offline approach). In the offline approach, the problem is replaced by its empirical counterpart (the empirical risk minimization problem). The natural question is how to define the pro…
This paper focuses on designing a unified approach for computing the projection onto the intersection of an $\ell_1$ ball/sphere and an $\ell_2$ ball/sphere. We show that the major computational efforts of solving these problems all rely on finding the root of the same piecewisely quadratic function, and then propose a unified numerical method to compute the root. In particular, we design breakpoint search methods with/without sorting incorporated with bisection, secant and Newton methods to find the interval containing the root, on which the root has a closed form. It can be shown that our pr…
Joint sparsity offers powerful structural cues for feature selection, especially for variables that are expected to demonstrate a "grouped" behavior. Such behavior is commonly modeled via group-lasso, multitask lasso, and related methods where feature selection is effected via mixed-norms. Several mixed-norm based sparse models have received substantial attention, and for some cases efficient algorithms are also available. Surprisingly, several constrained sparse models seem to be lacking scalable algorithms. We address this deficiency by presenting batch and online (stochastic-gradient) optim…
Tel Aviv University（特拉维夫大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）
We consider linear prediction with a convex Lipschitz loss, or more generally, stochastic convex optimization problems of generalized linear form, i.e.~where each instantaneous loss is a scalar convex function of a linear function. We show that in this setting, early stopped Gradient Descent (GD), without any explicit regularization or projection, ensures excess error at most $ε$ (compared to the best possible with unit Euclidean norm) with an optimal, up to logarithmic factors, sample complexity of $\tilde{O}(1/ε^2)$ and only $\tilde{O}(1/ε^2)$ iterations. This contrasts with general stochast…
This paper considers projection and convolution operations for integrally convex functions, which constitute a fundamental function class in discrete convex analysis. It is shown that the class of integrally convex functions is stable under projection, and this is also the case with the subclasses of integrally convex functions satisfying local or global discrete midpoint convexity. As is known in the literature, the convolution of two integrally convex functions may possibly fail to be integrally convex. We show that the convolution of an integrally convex function with a separable convex fun…
This article studies optional and predictable projections of integrands and convex-valued stochastic processes. The existence and uniqueness are shown under general conditions that are analogous to those for conditional expectations of integrands and random sets. In the convex case, duality correspondences between the projections and projections of epigraphs are given. These results are used to study projections of set-valued integrands. Consistently with the general theory of stochastic processes, projections are not constructed using reference measures on the optional and predictable sigma-a…
Technion - Israel Institute of Technology（以色列理工学院）
Smooth convex minimization over the unit trace-norm ball is an important optimization problem in machine learning, signal processing, statistics and other fields, that underlies many tasks in which one wishes to recover a low-rank matrix given certain measurements. While first-order methods for convex optimization enjoy optimal convergence rates, they require in worst-case to compute a full-rank SVD on each iteration, in order to compute the projection onto the trace-norm ball. These full-rank SVD computations however prohibit the application of such methods to large problems. A simple and nat…
Towards An Efficient Approach for the Nonconvex $\ell_p$ Ball Projection: Algorithm and Analysis
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Center for Applied Mathematics of Henan Province, Henan University（河南大学河南省应用数学中心）；Department of Mathematics, University of Washington（华盛顿大学数学系）
In this letter we revisit the famous heavy ball method and study its global convergence for a class of non-convex problems with sector-bounded gradient. We characterize the parameters that render the method globally convergent and yield the best $R$-convergence factor. We show that for this family of functions, this convergence factor is superior to the factor obtained from the triple momentum method.
A spectrahedron is a set defined by a linear matrix inequality. A projection of a spectrahedron is often called a semidefinitely representable set. We show that the convex hull of a finite union of such projections is again a projection of a spectrahedron. This improves upon the result of Helton and Nie, who prove the same result in the case of bounded sets.
We derive lower bounds on the black-box oracle complexity of large-scale smooth convex minimization problems, with emphasis on minimizing smooth (with Holder continuous, with a given exponent and constant, gradient) convex functions over high-dimensional ||.||_p-balls, 1<=p<=\infty. Our bounds turn out to be tight (up to logarithmic in the design dimension factors), and can be viewed as a substantial extension of the existing lower complexity bounds for large-scale convex minimization covering the nonsmooth case and the 'Euclidean' smooth case (minimization of convex functions with Lipschitz c…
In this paper, we study the rate of convergence of the cyclic projection algorithm applied to finitely many basic semi-algebraic convex sets. We establish an explicit convergence rate estimate which relies on the maximum degree of the polynomials that generate the basic semi-algebraic convex sets and the dimension of the underlying space. We achieve our results by exploiting the algebraic structure of the basic semi-algebraic convex sets.
Near-Linear Time Projection onto the $\ell_{1,\infty}$ Ball; Application to Sparse Autoencoders
Université Côte d’Azur（蔚蓝海岸大学）；CNRS（法国国家科学研究中心）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；SDAIA-KAUST AI（SDAIA-KAUST人工智能中心）
Technion - Israel Institute of Technology（以色列理工学院）
Projection-free optimization algorithms, which are mostly based on the classical Frank-Wolfe method, have gained significant interest in the machine learning community in recent years due to their ability to handle convex constraints that are popular in many applications, but for which computing projections is often computationally impractical in high-dimensional settings, and hence prohibit the use of most standard projection-based methods. In particular, a significant research effort was put on projection-free methods for online learning. In this paper we revisit the Online Frank-Wolfe (OFW)…
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Washington（华盛顿大学）；Tongji University（同济大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this paper, we propose a novel primal-dual inexact gradient projection method for nonlinear optimization problems with convex-set constraint. This method only needs inexact computation of the projections onto the convex set for each iteration, consequently reducing the computational cost for projections per iteration. This feature is attractive especially for solving problems where the projections are computationally not easy to calculate. Global convergence guarantee and O(1/k) ergodic convergence rate of the optimality residual are provided under loose assumptions. We apply our proposed s…
A new deconvolution algorithm based on orthogonal projections onto the epigraph set of a convex cost function is presented. In this algorithm, the dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. As the utilized cost function is a convex function in $R^N$, the corresponding epigraph set is also a convex set in $R^{N+1}$. The deconvolution algorithm starts with an arbitrary initial estimate in $R^{N+1}$. At each step of the iterative algorithm, first deconvolution projections are performed onto the epigraphs, later an orthogonal pro…
Bilkent University（比尔肯特大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
A new signal processing framework based on making orthogonal Projections onto the Epigraph Set of a Convex cost function (PESC) is developed. In this way it is possible to solve convex optimization problems using the well-known Projections onto Convex Set (POCS) approach. In this algorithm, the dimension of the minimization problem is lifted by one and a convex set corresponding to the epigraph of the cost function is defined. If the cost function is a convex function in $R^N$, the corresponding epigraph set is also a convex set in R^{N+1}. The PESC method provides globally optimal solutions f…
Two new optimization techniques based on projections onto convex space (POCS) framework for solving convex optimization problems are presented. The dimension of the minimization problem is lifted by one and sets corresponding to the cost function are defined. If the cost function is a convex function in R^N the corresponding set is also a convex set in R^{N+1}. The iterative optimization approach starts with an arbitrary initial estimate in R^{N+1} and an orthogonal projection is performed onto one of the sets in a sequential manner at each step of the optimization problem. The method provides…
Convex optimizers have known many applications as differentiable layers within deep neural architectures. One application of these convex layers is to project points into a convex set. However, both forward and backward passes of these convex layers are significantly more expensive to compute than those of a typical neural network. We investigate in this paper whether an inexact, but cheaper projection, can drive a descent algorithm to an optimum. Specifically, we propose an interpolation-based projection that is computationally cheap and easy to compute given a convex, domain defining, functi…
We prove that for any compact set B in R^d and for any epsilon >0 there is a finite subset X of B of |X|=d^{O(1/epsilon^2)} points such that the maximum absolute value of any linear function ell: R^d --> R on X approximates the maximum absolute value of ell on B within a factor of epsilon sqrt{d}. We also discuss approximations of convex bodies by projections of spectrahedra, that is, by projections of sections of the cone of positive semidefinite matrices by affine subspaces.
