Randomized iterative algorithms have attracted much attention in recent years because they can approximately solve large-scale linear systems of equations without accessing the entire coefficient matrix. In this paper, we propose two novel pseudoinverse-free randomized block iterative algorithms for solving consistent and inconsistent linear systems. The proposed algorithms require two user-defined random matrices: one for row sampling and the other for column sampling. We can recover the well-known doubly stochastic Gauss--Seidel, randomized Kaczmarz, randomized coordinate descent, and randomized extended Kaczmarz algorithms by choosing appropriate random matrices used in our algorithms. Because our algorithms allow for a much wider selection of these two random matrices, a number of new specific algorithms can be obtained. We prove the linear convergence in the mean square sense of our algorithms. Numerical experiments for linear systems with synthetic and real-world coefficient matrices demonstrate the efficiency of some special cases of our algorithms.
Consider solving large sparse range symmetric singular linear systems $ A {\bf x}= {\bf b} $ which arise, for instance, in the discretization of convection diffusion equations with periodic boundary conditions, and partial differential equations for electromagnetic fields using the edge-based finite element method. In theory, the Generalized Minimal Residual (GMRES) method converges to the least squares solution for inconsistent systems if the coefficient matrix $A$ is range symmetric, i.e. $ {\rm R}(A)= {\rm R}(A^{ \rm T } )$, where $ {\rm R}(A)$ is the range space of $A$. We derived the nece…
Randomized iterative algorithms have recently been proposed to solve large-scale linear systems. In this paper, we present a simple randomized extended block Kaczmarz algorithm that exponentially converges in the mean square to the unique minimum $\ell_2$-norm least squares solution of a given linear system of equations. The proposed algorithm is pseudoinverse-free and therefore different from the projection-based randomized double block Kaczmarz algorithm of Needell, Zhao, and Zouzias. We emphasize that our method works for all types of linear systems (consistent or inconsistent, overdetermin…
We investigate the randomized Kaczmarz method that adaptively updates the stepsize using readily available information for solving inconsistent linear systems. A novel geometric interpretation is provided which shows that the proposed method can be viewed as an orthogonal projection method in some sense. We prove that this method converges linearly in expectation to the unique minimum Euclidean norm least-squares solution of the linear system, and provide a tight upper bound for the convergence of the proposed method. Numerical experiments are also given to illustrate the theoretical results.
With a greedy strategy to construct control index set of coordinates firstly and then choosing the corresponding column submatrix in each iteration, we present a greedy block Gauss-Seidel (GBGS) method for solving large linear least squares problem. Theoretical analysis demonstrates that the convergence factor of the GBGS method can be much smaller than that of the greedy randomized coordinate descent (GRCD) method proposed recently in the literature. On the basis of the GBGS method, we further present a pseudoinverse-free greedy block Gauss-Seidel method, which doesn't need to calculate the M…
We introduce a new class of preconditioners to enable flexible GMRES to find a least-squares solution, and potentially the pseudoinverse solution, of large-scale sparse, asymmetric, singular, and potentially inconsistent systems. We develop the preconditioners based on a new observation that generalized inverses (i.e., $\boldsymbol{A}^{g}\in\{\boldsymbol{G}\mid\boldsymbol{A}\boldsymbol{G}\boldsymbol{A}=\boldsymbol{A}\}$) enable the preconditioned Krylov subspaces to converge in a single step. We then compute an approximate generalized inverse (AGI) efficiently using a hybrid incomplete factori…
Randomized SVD has become an extremely successful approach for efficiently computing a low-rank approximation of matrices. In particular the paper by Halko, Martinsson, and Tropp (SIREV 2011) contains extensive analysis, and has made it a very popular method. The typical complexity for a rank-$r$ approximation of $m\times n$ matrices is $O(mn\log n+(m+n)r^2)$ for dense matrices. The classical Nystr{ö}m method is much faster, but applicable only to positive semidefinite matrices. This work studies a generalization of Nystr{ö}m method applicable to general matrices, and shows that (i) it has nea…
A Tensor Greedy Double-Block Extended Kaczmarz Method for Inconsistent Tensor Linear Systems under the t-product
African Institute of Mathematical Sciences（非洲数学科学学院）；African Master’s of Machine Intelligence（非洲机器学习硕士项目）；Institute for Analysis and Algebra, TU Braunschweig（不伦瑞克工业大学分析与代数研究所）
Refined upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms
School of Mathematical Sciences and Fujian Provincial Key Laboratory of Mathematical Modeling and High-Performance Scientific Computing, Xiamen University（厦门大学数学科学学院、福建省数学建模与高性能科学计算重点实验室）
The randomized Kaczmarz (RK) method is an iterative method for approximating the least-squares solution of large linear systems of equations. The standard RK method uses sequential updates, making parallel computation difficult. Here, we study a parallel version of RK where a weighted average of independent updates is used. We analyze the convergence of RK with averaging and demonstrate its performance empirically. We show that as the number of threads increases, the rate of convergence improves and the convergence horizon for inconsistent systems decreases.
The M-P (Moore-Penrose) pseudoinverse has as a key application the computation of least-squares solutions of inconsistent systems of linear equations. Irrespective of whether a given input matrix is sparse, its M-P pseudoinverse can be dense, potentially leading to high computational burden, especially when we are dealing with high-dimensional matrices. The M-P pseudoinverse is uniquely characterized by four properties, but only two of them need to be satisfied for the computation of least-squares solutions. Fampa and Lee (2018) and Xu, Fampa, Lee, and Ponte (2019) propose local-search procedu…
We provide a complete characterization of the randomized Kaczmarz algorithm (RKA) for inconsistent linear systems. The Kaczmarz algorithm, known in some fields as the algebraic reconstruction technique, is a classical method for solving large-scale overdetermined linear systems through a sequence of projection operators; the randomized Kaczmarz algorithm is a recent proposal by Strohmer and Vershynin to randomize the sequence of projections in order to guarantee exponential convergence (in mean square) to the solutions. A flurry of work followed this development, with renewed interest in the a…
The Kaczmarz method is an iterative projection scheme for solving con-sistent system $Ax = b$. It is later extended to the inconsistent and ill-posed linear problems. But the classical Kaczmarz method is sensitive to the correlation of the adjacent equations. In order to reduce the impact of correlation on the convergence rate, the randomized Kaczmarz method and randomized block Kaczmarz method are proposed, respectively. In the current literature, the error estimate results of these methods are established based on the error $\|x_k-x_*\|_2$, where $x_*$ is the solution of linear system $Ax=b$…
In applications like medical imaging, error correction, and sensor networks, one needs to solve large-scale linear systems that may be corrupted by a small number of arbitrarily large corruptions. We consider solving such large-scale systems of linear equations $A\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector $\mathbf{b}$. With this as our motivating example, we develop an approach for this setting that allows detection of the corrupted entries and thus convergence to the "true" solution of the original system. We provide analytical justification for …
Randomized regularized Kaczmarz algorithms have recently been proposed to solve tensor recovery models with {\it consistent} linear measurements. In this work, we propose a novel algorithm based on the randomized extended Kaczmarz algorithm (which converges linearly in expectation to the unique minimum norm least squares solution of a linear system) for tensor recovery models with {\it inconsistent} linear measurements. We prove the linear convergence in expectation of our algorithm. Numerical experiments on a tensor least squares problem and a sparse tensor recovery problem are given to illus…
This paper investigates the randomized version of the Kaczmarz method to solve linear systems in the case where the adjoint of the system matrix is not exact---a situation we refer to as "mismatched adjoint". We show that the method may still converge both in the over- and underdetermined consistent case under appropriate conditions, and we calculate the expected asymptotic rate of linear convergence. Moreover, we analyze the inconsistent case and obtain results for the method with mismatched adjoint as for the standard method. Finally, we derive a method to compute optimized probabilities for…
The Extended Randomized Kaczmarz method is a well known iterative scheme which can find the Moore-Penrose inverse solution of a possibly inconsistent linear system and requires only one additional column of the system matrix in each iteration in comparison with the standard randomized Kaczmarz method. Also, the Sparse Randomized Kaczmarz method has been shown to converge linearly to a sparse solution of a consistent linear system. Here, we combine both ideas and propose an Extended Sparse Randomized Kaczmarz method. We show linear expected convergence to a sparse least squares solution in the …
We propose a simple doubly stochastic block Gauss--Seidel algorithm for solving linear systems of equations. By varying the row partition parameter and the column partition parameter of the coefficient matrix, we recover the Landweber algorithm, the randomized Kaczmarz algorithm, the randomized Gauss--Seidel algorithm, and the doubly stochastic Gauss--Seidel algorithm. For general (consistent or inconsistent) linear systems, we show the exponential convergence of the {\it norms of the expected iterates} via exact formulas. For consistent linear systems, we prove the exponential convergence of …
The standard randomized sparse Kaczmarz (RSK) method is an algorithm to compute sparse solutions of linear systems of equations and uses sequential updates, and thus, does not take advantage of parallel computations. In this work, we introduce a parallel (mini batch) version of RSK based on averaging several Kaczmarz steps. Naturally, this method allows for parallelization and we show that it can also leverage large over-relaxation. We prove linear expected convergence and show that, given that parallel computations can be exploited, the method provably provides faster convergence than the sta…
Worth Their Weight: Randomized and Regularized Block Kaczmarz Algorithms without Preprocessing
Department of Mathematics, UC Berkeley（数学系，加州大学伯克利分校）；Applied Mathematics and Computational Research Division, Lawrence Berkeley National Laboratory（应用数学与计算研究部，劳伦斯伯克利国家实验室）
College of Computing, Mohammed VI Polytechnic University, Morocco（穆罕默德六世理工大学计算机学院）；Department of Mathematics, University of Central Florida, USA（中央佛罗里达大学数学系）；Department of Mathematics, University of California, Irvine, USA（加州大学伊里布分校数学系）
