The famous greedy randomized Kaczmarz (GRK) method uses the greedy selection rule on maximum distance to determine a subset of the indices of working rows. In this paper, with the greedy selection rule on maximum residual, we propose the greedy randomized Motzkin-Kaczmarz (GRMK) method for linear systems. The block version of the new method is also presented. We analyze the convergence of the two methods and provide the corresponding convergence factors. Extensive numerical experiments show that the GRMK method has almost the same performance as the GRK method for dense matrices and the former performs better in computing time for some sparse matrices, and the block versions of the GRMK and GRK methods always have almost the same performance.
In this paper, for solving large-scale nonlinear equations we propose a nonlinear sampling Kaczmarz-Motzkin (NSKM) method. Based on the local tangential cone condition and the Jensen's inequality, we prove convergence of our method with two different assumptions. Then, for solving nonlinear equations with the convex constraints we present two variants of the NSKM method: the projected sampling Kaczmarz-Motzkin (PSKM) method and the accelerated projected sampling Kaczmarz-Motzkin (APSKM) method. With the use of the nonexpansive property of the projection and the convergence of the NSKM method, …
By introducing a subsampling strategy, we propose a randomized block Kaczmarz-Motzkin method for solving linear systems. Such strategy not only determines the block size, but also combines and extends two famous strategies, i.e., randomness and greed, and hence can inherit their advantages. Theoretical analysis shows that the proposed method converges linearly in expectation to the least-Euclidean-norm solution. Several numerical examples are reported to verify the efficiency and feasibility of the new method.
The randomized sparse Kaczmarz method was recently proposed to recover sparse solutions of linear systems. In this work, we introduce a greedy variant of the randomized sparse Kaczmarz method by employing the sampling Kaczmarz-Motzkin method, and prove its linear convergence in expectation with respect to the Bregman distance in the noiseless and noisy cases. This greedy variant can be viewed as a unification of the sampling Kaczmarz-Motzkin method and the randomized sparse Kaczmarz method, and hence inherits the merits of these two methods. Numerically, we report a couple of experimental resu…
The sampling Kaczmarz-Motzkin (SKM) method is a generalization of the randomized Kaczmarz and Motzkin methods. It first samples some rows of coefficient matrix randomly to build a set and then makes use of the maximum violation criterion within this set to determine a constraint. Finally, it makes progress by enforcing this single constraint. In this paper, on the basis of the framework of the SKM method and considering the greedy strategies, we present two block sampling Kaczmarz-Motzkin methods for consistent linear systems. Specifically, we also first sample a subset of rows of coefficient …
Stochastic iterative algorithms have gained recent interest in machine learning and signal processing for solving large-scale systems of equations, $Ax=b$. One such example is the Randomized Kaczmarz (RK) algorithm, which acts only on single rows of the matrix $A$ at a time. While RK randomly selects a row of $A$ to work with, Motzkin's Method (MM) employs a greedy row selection. Connections between the two algorithms resulted in the Sampling Kaczmarz-Motzkin (SKM) algorithm which samples a random subset of $β$ rows of $A$ and then greedily selects the best row of the subset. Despite their var…
With a quite different way to determine the working rows, we propose a novel greedy Kaczmarz method for solving consistent linear systems. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the greedy randomized Kaczmarz method and the relaxed greedy randomized Kaczmarz method introduced recently by Bai and Wu [Z.Z. BAI AND W.T. WU, On greedy randomized Kaczmarz method for solving large sparse linear systems, SIAM J. Sci. Comput., 40 (2018), pp. A592--A606; Z.Z. BAI AND W.T. WU, On relaxed greedy randomized Kaczmar…
The Kaczmarz method is an iterative algorithm for solving systems of linear equalities and inequalities, that iteratively projects onto these constraints. Recently, Strohmer and Vershynin [J. Fourier Anal. Appl., 15(2):262-278, 2009] gave a non-asymptotic convergence rate analysis for this algorithm, spurring numerous extensions and generalizations of the Kaczmarz method. Rather than the randomized selection rule analyzed in that work, in this paper we instead discuss greedy and approximate greedy selection rules. We show that in some applications the computational costs of greedy and random s…
For solving large-scale consistent linear system, we combine two efficient row index selection strategies with Kaczmarz-type method with oblique projection, and propose a greedy randomized Kaczmarz method with oblique projection (GRKO) and the maximal weighted residual Kaczmarz method with oblique projection (MWRKO) . Through those method, the number of iteration steps and running time can be reduced to a greater extent to find the least-norm solution, especially when the rows of matrix A are close to linear correlation. Theoretical proof and numerical results show that GRKO method and MWRKO m…
A class of fast greedy block Kaczmarz methods combined with general greedy strategy and average technique are proposed for solving large consistent linear systems. Theoretical analysis of the convergence of the proposed method is given in detail. Numerical experiments show that the proposed methods are efficient and faster than the existing methods.
The nonlinear Kaczmarz method was recently proposed to solve the system of nonlinear equations. In this paper, we first discuss two greedy selection rules, i.e., the maximum residual and maximum distance rules, for the nonlinear Kaczmarz iteration. Then, based on them, two kinds of greedy randomized sampling methods are presented. Further, we also devise four corresponding greedy randomized block methods, i.e., the multiple samples-based methods. The linear convergence in expectation of all the proposed methods is proved. Numerical results show that, in some applications including brown almost…
To solve nonlinear problems, we construct two kinds of greedy capped nonlinear Kaczmarz methods by setting a capped threshold and introducing an effective probability criterion for selecting a row of the Jacobian matrix. The capped threshold and probability criterion are mainly determined by the maximum residual and maximum distance rules. The block versions of the new methods are also presented. We provide the convergence analysis of these methods and their numerical results behave quite well.
A Tensor Greedy Double-Block Extended Kaczmarz Method for Inconsistent Tensor Linear Systems under the t-product
African Institute of Mathematical Sciences（非洲数学科学学院）；African Master’s of Machine Intelligence（非洲机器学习硕士项目）；Institute for Analysis and Algebra, TU Braunschweig（不伦瑞克工业大学分析与代数研究所）
The Kaczmarz method is a popular iterative scheme for solving large-scale linear systems. The randomized Kaczmarz method (RK) greatly improves the convergence rate of the Kaczmarz method, by using the rows of the coefficient matrix in random order rather than in their given order. An obvious disadvantage of the randomized Kaczmarz method is its probability criterion for selecting the active or working rows in the coefficient matrix. In [{\sc Z.Z. Bai, W. Wu}, {\em On greedy randomized Kaczmarz method for solving large sparse linear systems}, SIAM Journal on Scientific Computing, 2018, 40: A592…
Recently proposed adaptive Sketch & Project (SP) methods connect several well-known projection methods such as Randomized Kaczmarz (RK), Randomized Block Kaczmarz (RBK), Motzkin Relaxation (MR), Randomized Coordinate Descent (RCD), Capped Coordinate Descent (CCD), etc. into one framework for solving linear systems. In this work, we first propose a Stochastic Steepest Descent (SSD) framework that connects SP methods with the well-known Steepest Descent (SD) method for solving positive-definite linear system of equations. We then introduce two greedy sampling strategies in the SSD framework that…
The sketch-and-project, as a general archetypal algorithm for solving linear systems, unifies a variety of randomized iterative methods such as the randomized Kaczmarz and randomized coordinate descent. However, since it aims to find a least-norm solution from a linear system, the randomized sparse Kaczmarz can not be included. This motivates us to propose a more general framework, called sketched Bregman projection (SBP) method, in which we are able to find solutions with certain structures from linear systems. To generalize the concept of adaptive sampling to the SBP method, we show how the …
We propose using greedy and randomized Kaczmarz inner-iterations as preconditioners for the right-preconditioned flexible GMRES method to solve consistent linear systems, with a parameter tuning strategy for adjusting the number of inner iterations and the relaxation parameter. We also present theoretical justifications of the right-preconditioned flexible GMRES for solving consistent linear systems. Numerical experiments on overdetermined and underdetermined linear systems show that the proposed method is superior to the GMRES method preconditioned by NE-SOR inner iterations in terms of total…
For solving the large-scale linear system by iteration methods, we utilize the Petrov-Galerkin conditions and relaxed greedy index selection technique and provide two relaxed greedy deterministic row (RGDR) and column (RGDC) iterative methods, in which one special case of RGDR reduces to the fast deterministic block Kaczmarz method proposed in Chen and Huang (Numer. Algor., 89: 1007-1029, 2021). Our convergence analyses reveal that the resulting algorithms all have the linear convergence rates, which are bounded by the explicit expressions. Numerical examples show that the proposed algorithms …
We develop two greedy sampling rules for the Sketch & Project method for solving linear feasibility problems. The proposed greedy sampling rules generalize the existing max-distance sampling rule and uniform sampling rule and generate faster variants of Sketch & Project methods. We also introduce greedy capped sampling rules that improve the existing capped sampling rules. Moreover, we incorporate the so-called heavy ball momentum technique to the proposed greedy Sketch & Project method. By varying the parameters such as sampling rules, sketching vectors; we recover several well-known algorith…
We extend results known for the randomized Gauss-Seidel and the Gauss-Southwell methods for the case of a Hermitian and positive definite matrix to certain classes of non-Hermitian matrices. We obtain convergence results for a whole range of parameters describing the probabilities in the randomized method or the greedy choice strategy in the Gauss-Southwell-type methods. We identify those choices which make our convergence bounds best possible. Our main tool is to use weighted l1-norms to measure the residuals. A major result is that the best convergence bounds that we obtain for the expected …
The randomized projection (RP) method is a simple iterative scheme for solving linear feasibility problems and has recently gained popularity due to its speed and low memory requirement. This paper develops an accelerated variant of the standard RP method by using two ingredients: the greedy probability criterion and the average block approach, and obtains a greedy randomized average block projection (GRABP) method for solving large-scale systems of linear inequalities. We prove that this method converges linearly in expectation under different choices of extrapolated stepsizes. Numerical expe…
Solving large-scale systems of nonlinear equations/inequalities is a fundamental problem in computing and optimization. In this paper, we propose a generic successive projection (SP) framework for this problem. The SP sequentially projects the current iterate onto the constraint set corresponding to each nonlinear (in)equality. It extends von Neumann's alternating projection for finding a point in the intersection of two linear subspaces, Bregman's method for finding a common point of convex sets and the Kaczmarz method for solving systems of linear equations to the more general case of multip…
