In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization. It is well understood that the condition number of the problem, i.e., the ratio of the Lipschitz constant to the strong convexity modulus, has a harsh effect on the convergence of the first-order optimization methods. Therefore, minimizing a small regularized loss for achieving good generalization performance, yielding an ill conditioned problem, becomes the bottleneck for big data problems. We provide a theory on data preconditioning for regularized loss minimization. In particular, our analysis exhibits an appropriate data preconditioner and characterizes the conditions on the loss function and on the data under which data preconditioning can reduce the condition number and therefore boost the convergence for minimizing the regularized loss. To make the data preconditioning practically useful, we endeavor to employ and analyze a random sampling approach to efficiently compute the preconditioned data. The preliminary experiments validate our theory.
In this paper we present general-purpose preconditioners for regularized augmented systems arising from optimization problems, and their corresponding normal equations. We discuss positive definite preconditioners, suitable for CG and MINRES. We consider "sparsifications" which avoid situations in which eigenvalues of the preconditioned matrix may become complex. Special attention is given to systems arising from the application of regularized interior point methods to linear or nonlinear convex programming problems.
Recent quasi-optimal error estimates for the finite element approximation of total-variation regularized minimization problems require the existence of a Lipschitz continuous dual solution. We discuss the validity of this condition and devise numerical methods using locally refined meshes that lead to improved convergence rates despite the occurrence of discontinuities. It turns out that nearly linear convergence is possible on suitably constructed meshes.
Recent quasi-optimal error estimates for the finite element approximation of total-variation regularized minimization problems using the Crouzeix--Raviart finite element require the existence of a Lipschitz continuous dual solution, which is not generally given. We provide analytic proofs showing that the Lipschitz continuity of a dual solution is not necessary, in general. Using the Lipschitz truncation technique, we, in addition, derive error estimates that depend directly on the Sobolev regularity of a given dual solution.
In this work, in the context of Linear and Quadratic Programming, we interpret Primal Dual Regularized Interior Point Methods (PDR-IPMs) in the framework of the Proximal Point Method. The resulting Proximal Stabilized IPM (PS-IPM) is strongly supported by theoretical results concerning convergence and the rate of convergence, and can handle degenerate problems. Moreover, in the second part of this work, we analyse the interactions between the regularization parameters and the computational foot-print of the linear algebra routines used to solve the Newton linear systems. In particular, when th…
In this paper, we theoretically investigate the low-rank matrix recovery problem in the context of the unconstrained regularized nuclear norm minimization (RNNM) framework. Our theoretical findings show that, the RNNM method is able to provide a robust recovery of any matrix $X$ (not necessary to be exactly low-rank) from its few noisy measurements $\textbf{b}=\mathcal{A}(X)+\textbf{n}$ with a bounded constraint $\|\textbf{n}\|_{2}\leqε$, provided that the $tk$-order restricted isometry constant (RIC) of $\mathcal{A}$ satisfies a certain constraint related to $t>0$. Specifically, the obtained …
In this paper, we address the efficient numerical solution of linear and quadratic programming problems, often of large scale. With this aim, we devise an infeasible interior point method, blended with the proximal method of multipliers, which in turn results in a primal-dual regularized interior point method. Application of this method gives rise to a sequence of increasingly ill-conditioned linear systems which cannot always be solved by factorization methods, due to memory and CPU time restrictions. We propose a novel preconditioning strategy which is based on a suitable sparsification of t…
We consider a linear iterative solver for large scale linearly constrained quadratic minimization problems that arise, for example, in optimization with PDEs. By a primal-dual projection (PDP) iteration, which can be interpreted and analysed as a gradient method on a quotient space, the given problem can be solved by computing sulutions for a sequence of constrained surrogate problems, projections onto the feasible subspaces, and Lagrange multiplier updates. As a major application we consider a class of optimization problems with PDEs, where PDP can be applied together with a projected cg meth…
Alternating minimization algorithms for graph regularized tensor completion
Huawei European Research Institute（华为欧洲研究院）；LISN, INRIA, Université Paris-Saclay（法国国家信息与自动化研究所，巴黎萨克雷大学）；LSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院科学与工程计算国家重点实验室）；Department of INMA, Université catholique de Louvain（鲁汶天主教大学）
This paper deals with regularized Newton methods, a flexible class of unconstrained optimization algorithms that is competitive with line search and trust region methods and potentially combines attractive elements of both. The particular focus is on combining regularization with limited memory quasi-Newton methods by exploiting the special structure of limited memory algorithms. Global convergence of regularization methods is shown under mild assumptions and the details of regularized limited memory quasi-Newton updates are discussed including their compact representations. Numerical results …
Parameter-free inexact block Schur complement preconditioning for linear poroelasticity under a hybrid Bernardi-Raugel and weak Galerkin finite element discretization
Nonmonotone Barzilai-Borwein Gradient Algorithm for $\ell_1$-Regularized Nonsmooth Minimization in Compressive Sensing
Institute of Applied Mathematics, College of Mathematics and Information Science, Henan University（河南大学数学与信息科学学院应用数学研究所）；National Center for Theoretical Sciences (South), National Cheng Kung University（台湾成功大学南方理论科学中心）；Department of Applied Mathematics, Hong Kong Polytechnic University（香港理工大学应用数学系）
We propose a new method for low-rank approximation of Moore-Penrose pseudoinverses (MPPs) of large-scale matrices using tensor networks. The computed pseudoinverses can be useful for solving or preconditioning of large-scale overdetermined or underdetermined systems of linear equations. The computation is performed efficiently and stably based on the modified alternating least squares (MALS) scheme using low-rank tensor train (TT) decompositions and tensor network contractions. The formulated large-scale optimization problem is reduced to sequential smaller-scale problems for which any standar…
Principal component analysis (PCA) is known to be sensitive to outliers, so that various robust PCA variants were proposed in the literature. A recent model, called REAPER, aims to find the principal components by solving a convex optimization problem. Usually the number of principal components must be determined in advance and the minimization is performed over symmetric positive semi-definite matrices having the size of the data, although the number of principal components is substantially smaller. This prohibits its use if the dimension of the data is large which is often the case in image …
This article discusses nonconforming finite element methods for convex minimization problems and systematically derives dual mixed formulations. Duality relations lead to simple error estimates that avoid an explicit treatment of nonconformity errors. A reconstruction formula provides the discrete solution of the dual problem via a simple postprocessing procedure which implies a strong duality relation and is of interest in a posteriori error estimation. The framework applies to differentiable and nonsmooth problems, examples include $p$-Laplace, total-variation regularized, and obstacle probl…
Standard regularization methods that are used to compute solutions to ill-posed inverse problems require knowledge of the forward model. In many real-life applications, the forward model is not known, but training data is readily available. In this paper, we develop a new framework that uses training data, as a substitute for knowledge of the forward model, to compute an optimal low-rank regularized inverse matrix directly, allowing for very fast computation of a regularized solution. We consider a statistical framework based on Bayes and empirical Bayes risk minimization to analyze theoretica…
We consider a special nonconvex quartic minimization problem over a single spherical constraint, which includes the discretized energy functional minimization problem of non-rotating Bose-Einstein condensates (BECs) as one of the important applications. Such a problem is studied by exploiting its characterization as a nonlinear eigenvalue problem with eigenvector nonlinearity (NEPv), which admits a unique nonnegative eigenvector, and this eigenvector is exactly the global minimizer to the quartic minimization. With these properties, any algorithm converging to the nonnegative stationary point …
In this paper, we consider a primal-dual domain decomposition method for total variation regularized problems appearing in mathematical image processing. The model problem is transformed into an equivalent constrained minimization problem by tearing-and-interconnecting domain decomposition. Then, the continuity constraints on the subdomain interfaces are treated by introducing Lagrange multipliers. The resulting saddle point problem is solved by the first order primal-dual algorithm. We apply the proposed method to image denoising, inpainting, and segmentation problems with either $L^2$-fideli…
We propose and analyze a new numerical method for computing the ground state of the modified Gross-Pitaevskii equation for modeling the Bose-Einstein condensate with a higher order interaction by adapting the density function formulation and the accelerated projected gradient method. By reformulating the energy functional $E(ϕ)$ with $ϕ$, the wave function, in terms of the density $ρ=|ϕ|^2$, the original non-convex minimization problem for defining the ground state is then reformulated to a convex minimization problem. In order to overcome the semi-smoothness of the function $\sqrtρ$ in the ki…
In large-scale unconstrained optimization algorithms such as limited memory BFGS (LBFGS), a common subproblem is a line search minimizing the loss function along a descent direction. Commonly used line searches iteratively find an approximate solution for which the Wolfe conditions are satisfied, typically requiring multiple function and gradient evaluations per line search, which is expensive in parallel due to communication requirements. In this paper we propose a new line search approach for cases where the loss function is analytic, as in least squares regression, logistic regression, or l…
