With a quite different way to determine the working rows, we propose a novel greedy Kaczmarz method for solving consistent linear systems. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the greedy randomized Kaczmarz method and the relaxed greedy randomized Kaczmarz method introduced recently by Bai and Wu [Z.Z. BAI AND W.T. WU, On greedy randomized Kaczmarz method for solving large sparse linear systems, SIAM J. Sci. Comput., 40 (2018), pp. A592--A606; Z.Z. BAI AND W.T. WU, On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems, Appl. Math. Lett., 83 (2018), pp. 21--26] in term of the computing time.
The Kaczmarz method is an iterative algorithm for solving systems of linear equalities and inequalities, that iteratively projects onto these constraints. Recently, Strohmer and Vershynin [J. Fourier Anal. Appl., 15(2):262-278, 2009] gave a non-asymptotic convergence rate analysis for this algorithm, spurring numerous extensions and generalizations of the Kaczmarz method. Rather than the randomized selection rule analyzed in that work, in this paper we instead discuss greedy and approximate greedy selection rules. We show that in some applications the computational costs of greedy and random s…
For solving large-scale consistent linear system, we combine two efficient row index selection strategies with Kaczmarz-type method with oblique projection, and propose a greedy randomized Kaczmarz method with oblique projection (GRKO) and the maximal weighted residual Kaczmarz method with oblique projection (MWRKO) . Through those method, the number of iteration steps and running time can be reduced to a greater extent to find the least-norm solution, especially when the rows of matrix A are close to linear correlation. Theoretical proof and numerical results show that GRKO method and MWRKO m…
A class of fast greedy block Kaczmarz methods combined with general greedy strategy and average technique are proposed for solving large consistent linear systems. Theoretical analysis of the convergence of the proposed method is given in detail. Numerical experiments show that the proposed methods are efficient and faster than the existing methods.
The nonlinear Kaczmarz method was recently proposed to solve the system of nonlinear equations. In this paper, we first discuss two greedy selection rules, i.e., the maximum residual and maximum distance rules, for the nonlinear Kaczmarz iteration. Then, based on them, two kinds of greedy randomized sampling methods are presented. Further, we also devise four corresponding greedy randomized block methods, i.e., the multiple samples-based methods. The linear convergence in expectation of all the proposed methods is proved. Numerical results show that, in some applications including brown almost…
The famous greedy randomized Kaczmarz (GRK) method uses the greedy selection rule on maximum distance to determine a subset of the indices of working rows. In this paper, with the greedy selection rule on maximum residual, we propose the greedy randomized Motzkin-Kaczmarz (GRMK) method for linear systems. The block version of the new method is also presented. We analyze the convergence of the two methods and provide the corresponding convergence factors. Extensive numerical experiments show that the GRMK method has almost the same performance as the GRK method for dense matrices and the former…
To solve nonlinear problems, we construct two kinds of greedy capped nonlinear Kaczmarz methods by setting a capped threshold and introducing an effective probability criterion for selecting a row of the Jacobian matrix. The capped threshold and probability criterion are mainly determined by the maximum residual and maximum distance rules. The block versions of the new methods are also presented. We provide the convergence analysis of these methods and their numerical results behave quite well.
A Tensor Greedy Double-Block Extended Kaczmarz Method for Inconsistent Tensor Linear Systems under the t-product
African Institute of Mathematical Sciences（非洲数学科学学院）；African Master’s of Machine Intelligence（非洲机器学习硕士项目）；Institute for Analysis and Algebra, TU Braunschweig（不伦瑞克工业大学分析与代数研究所）
The Kaczmarz method is a popular iterative scheme for solving large-scale linear systems. The randomized Kaczmarz method (RK) greatly improves the convergence rate of the Kaczmarz method, by using the rows of the coefficient matrix in random order rather than in their given order. An obvious disadvantage of the randomized Kaczmarz method is its probability criterion for selecting the active or working rows in the coefficient matrix. In [{\sc Z.Z. Bai, W. Wu}, {\em On greedy randomized Kaczmarz method for solving large sparse linear systems}, SIAM Journal on Scientific Computing, 2018, 40: A592…
The randomized sparse Kaczmarz method was recently proposed to recover sparse solutions of linear systems. In this work, we introduce a greedy variant of the randomized sparse Kaczmarz method by employing the sampling Kaczmarz-Motzkin method, and prove its linear convergence in expectation with respect to the Bregman distance in the noiseless and noisy cases. This greedy variant can be viewed as a unification of the sampling Kaczmarz-Motzkin method and the randomized sparse Kaczmarz method, and hence inherits the merits of these two methods. Numerically, we report a couple of experimental resu…
Stochastic iterative algorithms have gained recent interest in machine learning and signal processing for solving large-scale systems of equations, $Ax=b$. One such example is the Randomized Kaczmarz (RK) algorithm, which acts only on single rows of the matrix $A$ at a time. While RK randomly selects a row of $A$ to work with, Motzkin's Method (MM) employs a greedy row selection. Connections between the two algorithms resulted in the Sampling Kaczmarz-Motzkin (SKM) algorithm which samples a random subset of $β$ rows of $A$ and then greedily selects the best row of the subset. Despite their var…
The sampling Kaczmarz-Motzkin (SKM) method is a generalization of the randomized Kaczmarz and Motzkin methods. It first samples some rows of coefficient matrix randomly to build a set and then makes use of the maximum violation criterion within this set to determine a constraint. Finally, it makes progress by enforcing this single constraint. In this paper, on the basis of the framework of the SKM method and considering the greedy strategies, we present two block sampling Kaczmarz-Motzkin methods for consistent linear systems. Specifically, we also first sample a subset of rows of coefficient …
In this paper, combining count sketch and maximal weighted residual Kaczmarz method, we propose a fast randomized algorithm for large overdetermined linear systems. Convergence analysis of the new algorithm is provided. Numerical experiments show that, for the same accuracy, our method behaves better in computing time compared with the state-of-the-art algorithm.
In this paper, for solving large-scale nonlinear equations we propose a nonlinear sampling Kaczmarz-Motzkin (NSKM) method. Based on the local tangential cone condition and the Jensen's inequality, we prove convergence of our method with two different assumptions. Then, for solving nonlinear equations with the convex constraints we present two variants of the NSKM method: the projected sampling Kaczmarz-Motzkin (PSKM) method and the accelerated projected sampling Kaczmarz-Motzkin (APSKM) method. With the use of the nonexpansive property of the projection and the convergence of the NSKM method, …
Recently proposed adaptive Sketch & Project (SP) methods connect several well-known projection methods such as Randomized Kaczmarz (RK), Randomized Block Kaczmarz (RBK), Motzkin Relaxation (MR), Randomized Coordinate Descent (RCD), Capped Coordinate Descent (CCD), etc. into one framework for solving linear systems. In this work, we first propose a Stochastic Steepest Descent (SSD) framework that connects SP methods with the well-known Steepest Descent (SD) method for solving positive-definite linear system of equations. We then introduce two greedy sampling strategies in the SSD framework that…
In this paper, an extension of Kaczmarz method, the Kaczmarz method with oblique projection (KO), is introduced and analyzed. Using this method, a number of iteration steps to solve the over-determined systems of linear equations are significantly reduced, and the the computing time is much saved, especially for those problems that contain some linear equations with near-linear correlation. Simultaneously, a randomized version--randomized Kaczmarz method with oblique projection (RKO) is established. The convergence proofs of these two methods are given and numerical experiments show the effect…
In this article a modified Levenberg-Marquardt method coupled with a Kaczmarz strategy for obtaining stable solutions of nonlinear systems of ill-posed operator equations is investigated. We show that the proposed method is a convergent regularization method. Numerical tests are presented for a non-linear inverse doping problem based on a bipolar model.
We propose using greedy and randomized Kaczmarz inner-iterations as preconditioners for the right-preconditioned flexible GMRES method to solve consistent linear systems, with a parameter tuning strategy for adjusting the number of inner iterations and the relaxation parameter. We also present theoretical justifications of the right-preconditioned flexible GMRES for solving consistent linear systems. Numerical experiments on overdetermined and underdetermined linear systems show that the proposed method is superior to the GMRES method preconditioned by NE-SOR inner iterations in terms of total…
The randomized Kaczmarz algorithm has received considerable attention recently because of its simplicity, speed, and the ability to approximately solve large-scale linear systems of equations. In this paper we propose randomized double and triple Kaczmarz algorithms to solve extended normal equations of the form $\bf A^\top Ax=A^\top b-c$. The proposed algorithms avoid forming $\bf A^\top A$ explicitly and work for {\it arbitrary} $\mbf A\in\mbbr^{m\times n}$ (full rank or rank deficient, $m\geq n$ or $m<n$). {\it Tight} upper bounds showing exponential convergence in the mean square sense of …
In this article we combine the projective Landweber method, recently proposed by the authors, with Kaczmarz's method for solving systems of non-linear ill-posed equations. The underlying assumption used in this work is the tangential cone condition. We show that the proposed iteration is a convergent regularization method. Numerical tests are presented for a non-linear inverse problem related to the Dirichlet-to-Neumann map, indicating a superior performance of the proposed method when compared with other well established iterations. Our preliminary investigation indicates that the resulting i…
Randomized iterative algorithms have recently been proposed to solve large-scale linear systems. In this paper, we present a simple randomized extended block Kaczmarz algorithm that exponentially converges in the mean square to the unique minimum $\ell_2$-norm least squares solution of a given linear system of equations. The proposed algorithm is pseudoinverse-free and therefore different from the projection-based randomized double block Kaczmarz algorithm of Needell, Zhao, and Zouzias. We emphasize that our method works for all types of linear systems (consistent or inconsistent, overdetermin…
A randomized Kaczmarz method was recently proposed for phase retrieval, which has been shown numerically to exhibit empirical performance over other state-of-the-art phase retrieval algorithms both in terms of the sampling complexity and in terms of computation time. While the rate of convergence has been studied well in the real case where the signals and measurement vectors are all real-valued, there is no guarantee for the convergence in the complex case. In fact, the linear convergence of the randomized Kaczmarz method for phase retrieval in the complex setting is left as a conjecture by T…
The sketch-and-project (SAP) framework for solving systems of linear equations has unified the theory behind popular projective iterative methods such as randomized Kaczmarz, randomized coordinate descent, and variants thereof. The randomized extended Kaczmarz (REK) method is a popular extension of randomized Kaczmarz for solving inconsistent systems, which has not yet been shown to lie within the SAP framework. In this work we show that, in a certain sense, REK may be expressed as the limit point of a family of SAP methods, but we argue that it is unlikely that REK can be translated into a SA…
Recovering a signal $x^\ast \in \mathbb{R}^n$ from a sequence of linear measurements is an important problem in areas such as computerized tomography and compressed sensing. In this work, we consider an online setting in which measurements are sampled one-by-one from some source distribution. We propose solving this problem with a variant of the Kaczmarz method with an additional heavy ball momentum term. A popular technique for solving systems of linear equations, recent work has shown that the Kaczmarz method also enjoys linear convergence when applied to random measurement models, however c…
The Kaczmarz method is an iterative projection scheme for solving con-sistent system $Ax = b$. It is later extended to the inconsistent and ill-posed linear problems. But the classical Kaczmarz method is sensitive to the correlation of the adjacent equations. In order to reduce the impact of correlation on the convergence rate, the randomized Kaczmarz method and randomized block Kaczmarz method are proposed, respectively. In the current literature, the error estimate results of these methods are established based on the error $\|x_k-x_*\|_2$, where $x_*$ is the solution of linear system $Ax=b$…
Solving linear systems of equations is a fundamental problem in mathematics. When the linear system is so large that it cannot be loaded into memory at once, iterative methods such as the randomized Kaczmarz method excel. Here, we extend the randomized Kaczmarz method to solve multi-linear (tensor) systems under the tensor-tensor t-product. We provide convergence guarantees for the proposed tensor randomized Kaczmarz that are analogous to those of the randomized Kaczmarz method for matrix linear systems. We demonstrate experimentally that the tensor randomized Kaczmarz method converges faster …
A Kaczmarz Algorithm for Solving Tree Based Distributed Systems of Equations
Electrical and Computer Engineering, Iowa State University, Ames, IA 50011（电子与计算机工程系，爱荷华州立大学，爱荷华州阿姆斯，IA 50011）；Department of Mathematics, Iowa State University, 396 Carver Hall, Ames, IA 50011（数学系，爱荷华州立大学，396号Carver大楼，爱荷华州阿姆斯，IA 50011）
By introducing a subsampling strategy, we propose a randomized block Kaczmarz-Motzkin method for solving linear systems. Such strategy not only determines the block size, but also combines and extends two famous strategies, i.e., randomness and greed, and hence can inherit their advantages. Theoretical analysis shows that the proposed method converges linearly in expectation to the least-Euclidean-norm solution. Several numerical examples are reported to verify the efficiency and feasibility of the new method.
