The randomized row method is a popular representative of the iterative algorithm because of its efficiency in solving the overdetermined and consistent systems of linear equations. In this paper, we present an extended randomized multiple row method to solve a given overdetermined and inconsistent linear system and analyze its computational complexities at each iteration. We prove that the proposed method can linearly converge in the mean square to the least-squares solution with a minimum Euclidean norm. Several numerical studies are presented to corroborate our theoretical findings. The real-world applications, such as image reconstruction and large noisy data fitting in computer-aided geometric design, are also presented for illustration purposes.
For solving large-scale consistent linear system, we combine two efficient row index selection strategies with Kaczmarz-type method with oblique projection, and propose a greedy randomized Kaczmarz method with oblique projection (GRKO) and the maximal weighted residual Kaczmarz method with oblique projection (MWRKO) . Through those method, the number of iteration steps and running time can be reduced to a greater extent to find the least-norm solution, especially when the rows of matrix A are close to linear correlation. Theoretical proof and numerical results show that GRKO method and MWRKO m…
The block Kaczmarz method is an iterative scheme for solving overdetermined least-squares problems. At each step, the algorithm projects the current iterate onto the solution space of a subset of the constraints. This paper describes a block Kaczmarz algorithm that uses a randomized control scheme to choose the subset at each step. This algorithm is the first block Kaczmarz method with an (expected) linear rate of convergence that can be expressed in terms of the geometric properties of the matrix and its submatrices. The analysis reveals that the algorithm is most effective when it is given a…
Randomized Row and Column Iterative Methods with a Quantum Computer
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）
This paper is about randomized iterative algorithms for solving a linear system of equations $X β= y$ in different settings. Recent interest in the topic was reignited when Strohmer and Vershynin (2009) proved the linear convergence rate of a Randomized Kaczmarz (RK) algorithm that works on the rows of $X$ (data points). Following that, Leventhal and Lewis (2010) proved the linear convergence of a Randomized Coordinate Descent (RCD) algorithm that works on the columns of $X$ (features). The aim of this paper is to simplify our understanding of these two algorithms, establish the direct relatio…
The kappa_SQ software package is designed to assist researchers working on randomized row sampling. The package contains a collection of Matlab functions along with a GUI that ties them all together and provides a platform for the user to perform experiments. In particular, kappa_SQ is designed to do experiments related to the two-norm condition number of a sampled matrix, $κ(SQ)$, where $S$ is a row sampling matrix and $Q$ is a tall and skinny matrix with orthonormal columns. Via a simple GUI, kappa_SQ can generate test matrices, perform various types of row sampling, measure $κ(SQ)$, calcula…
Deterministic and randomized, row-action and column-action linear solvers have become increasingly popular owing to their simplicity, low computational and memory complexities, and ease of composition with other techniques. Moreover, in order to achieve high-performance, such solvers must often be adapted to the given problem structure and to the hardware platform on which the problem will be solved. Unfortunately, determining whether such adapted solvers will converge to a solution has required equally unique analyses. As a result, adapted, reliable solvers are slow to be developed and deploy…
A class of restarted randomized surrounding methods are presented to accelerate the surrounding algorithms by restarted techniques for solving the linear equations. Theoretical analysis prove that the proposed method converges under the randomized row selection rule and the expectation convergence rate is also addressed. Numerical experiments further demonstrate that the proposed algorithms are efficient and outperform the existing method for over-determined and under-determined linear equation, as well as in the application of image processing.
We describe an asynchronous parallel variant of the randomized Kaczmarz (RK) algorithm for solving the linear system $Ax=b$. The analysis shows linear convergence and indicates that nearly linear speedup can be expected if the number of processors is bounded by a multiple of the number of rows in $A$.
For solving the large-scale linear system by iteration methods, we utilize the Petrov-Galerkin conditions and relaxed greedy index selection technique and provide two relaxed greedy deterministic row (RGDR) and column (RGDC) iterative methods, in which one special case of RGDR reduces to the fast deterministic block Kaczmarz method proposed in Chen and Huang (Numer. Algor., 89: 1007-1029, 2021). Our convergence analyses reveal that the resulting algorithms all have the linear convergence rates, which are bounded by the explicit expressions. Numerical examples show that the proposed algorithms …
The Kaczmarz method is a popular iterative scheme for solving large-scale linear systems. The randomized Kaczmarz method (RK) greatly improves the convergence rate of the Kaczmarz method, by using the rows of the coefficient matrix in random order rather than in their given order. An obvious disadvantage of the randomized Kaczmarz method is its probability criterion for selecting the active or working rows in the coefficient matrix. In [{\sc Z.Z. Bai, W. Wu}, {\em On greedy randomized Kaczmarz method for solving large sparse linear systems}, SIAM Journal on Scientific Computing, 2018, 40: A592…
The randomized Kaczmarz algorithm has received considerable attention recently because of its simplicity, speed, and the ability to approximately solve large-scale linear systems of equations. In this paper we propose randomized double and triple Kaczmarz algorithms to solve extended normal equations of the form $\bf A^\top Ax=A^\top b-c$. The proposed algorithms avoid forming $\bf A^\top A$ explicitly and work for {\it arbitrary} $\mbf A\in\mbbr^{m\times n}$ (full rank or rank deficient, $m\geq n$ or $m<n$). {\it Tight} upper bounds showing exponential convergence in the mean square sense of …
Randomized iterative algorithms have recently been proposed to solve large-scale linear systems. In this paper, we present a simple randomized extended block Kaczmarz algorithm that exponentially converges in the mean square to the unique minimum $\ell_2$-norm least squares solution of a given linear system of equations. The proposed algorithm is pseudoinverse-free and therefore different from the projection-based randomized double block Kaczmarz algorithm of Needell, Zhao, and Zouzias. We emphasize that our method works for all types of linear systems (consistent or inconsistent, overdetermin…
We extend results known for the randomized Gauss-Seidel and the Gauss-Southwell methods for the case of a Hermitian and positive definite matrix to certain classes of non-Hermitian matrices. We obtain convergence results for a whole range of parameters describing the probabilities in the randomized method or the greedy choice strategy in the Gauss-Southwell-type methods. We identify those choices which make our convergence bounds best possible. Our main tool is to use weighted l1-norms to measure the residuals. A major result is that the best convergence bounds that we obtain for the expected …
With a quite different way to determine the working rows, we propose a novel greedy Kaczmarz method for solving consistent linear systems. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the greedy randomized Kaczmarz method and the relaxed greedy randomized Kaczmarz method introduced recently by Bai and Wu [Z.Z. BAI AND W.T. WU, On greedy randomized Kaczmarz method for solving large sparse linear systems, SIAM J. Sci. Comput., 40 (2018), pp. A592--A606; Z.Z. BAI AND W.T. WU, On relaxed greedy randomized Kaczmar…
The Kaczmarz method for solving a linear system $Ax = b$ interprets such a system as a collection of equations $\left\langle a_i, x\right\rangle = b_i$, where $a_i$ is the $i-$th row of $A$, then picks such an equation and corrects $x_{k+1} = x_k + λa_i$ where $λ$ is chosen so that the $i-$th equation is satisfied. Convergence rates are difficult to establish. Assuming the rows to be normalized, $\|a_i\|_{\ell^2}=1$, Strohmer \& Vershynin established that if the order of equations is chosen at random, $\mathbb{E}~ \|x_k - x\|_{\ell^2}$ converges exponentially. We prove that if the $i-$th row i…
The indefinite least squares (ILS) problem is a generalization of the famous linear least squares problem. It minimizes an indefinite quadratic form with respect to a signature matrix. For this problem, we first propose an impressively simple and effective splitting (SP) method according to its own structure and prove that it converges 'unconditionally' for any initial value. Further, to avoid implementing some matrix multiplications and calculating the inverse of large matrix and considering the acceleration and efficiency of the randomized strategy, we develop two randomized iterative method…
We develop a randomized Newton's method for solving differential equations, based on a fully connected neural network discretization. In particular, the randomized Newton's method randomly chooses equations from the overdetermined nonlinear system resulting from the neural network discretization and solves the nonlinear system adaptively. We prove theoretically that the randomized Newton's method has a quadratic convergence locally. We also apply this new method to various numerical examples, from one- to high-dimensional differential equations, in order to verify its feasibility and efficienc…
Randomized iterative algorithms have attracted much attention in recent years because they can approximately solve large-scale linear systems of equations without accessing the entire coefficient matrix. In this paper, we propose two novel pseudoinverse-free randomized block iterative algorithms for solving consistent and inconsistent linear systems. The proposed algorithms require two user-defined random matrices: one for row sampling and the other for column sampling. We can recover the well-known doubly stochastic Gauss--Seidel, randomized Kaczmarz, randomized coordinate descent, and random…
We present a novel greedy Gauss-Seidel method for solving large linear least squares problem. This method improves the greedy randomized coordinate descent (GRCD) method proposed recently by Bai and Wu [Bai ZZ, and Wu WT. On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer Linear Algebra Appl. 2019;26(4):1--15], which in turn improves the popular randomized Gauss-Seidel method. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the GRCD method in term of the comput…
A randomized Kaczmarz method was recently proposed for phase retrieval, which has been shown numerically to exhibit empirical performance over other state-of-the-art phase retrieval algorithms both in terms of the sampling complexity and in terms of computation time. While the rate of convergence has been studied well in the real case where the signals and measurement vectors are all real-valued, there is no guarantee for the convergence in the complex case. In fact, the linear convergence of the randomized Kaczmarz method for phase retrieval in the complex setting is left as a conjecture by T…
The sketch-and-project (SAP) framework for solving systems of linear equations has unified the theory behind popular projective iterative methods such as randomized Kaczmarz, randomized coordinate descent, and variants thereof. The randomized extended Kaczmarz (REK) method is a popular extension of randomized Kaczmarz for solving inconsistent systems, which has not yet been shown to lie within the SAP framework. In this work we show that, in a certain sense, REK may be expressed as the limit point of a family of SAP methods, but we argue that it is unlikely that REK can be translated into a SA…
This paper investigates the randomized version of the Kaczmarz method to solve linear systems in the case where the adjoint of the system matrix is not exact---a situation we refer to as "mismatched adjoint". We show that the method may still converge both in the over- and underdetermined consistent case under appropriate conditions, and we calculate the expected asymptotic rate of linear convergence. Moreover, we analyze the inconsistent case and obtain results for the method with mismatched adjoint as for the standard method. Finally, we derive a method to compute optimized probabilities for…
We introduce a randomized algorithm, namely RCHOL, to construct an approximate Cholesky factorization for a given Laplacian matrix (a.k.a., graph Laplacian). From a graph perspective, the exact Cholesky factorization introduces a clique in the underlying graph after eliminating a row/column. By randomization, RCHOL only retains a sparse subset of the edges in the clique using a random sampling developed by Spielman and Kyng. We prove RCHOL is breakdown-free and apply it to solving large sparse linear systems with symmetric diagonally dominant matrices. In addition, we parallelize RCHOL based o…
