Refined upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms
School of Mathematical Sciences and Fujian Provincial Key Laboratory of Mathematical Modeling and High-Performance Scientific Computing, Xiamen University（厦门大学数学科学学院、福建省数学建模与高性能科学计算重点实验室）
The randomized extended Kaczmarz and Gauss-Seidel algorithms have attracted much attention because of their ability to treat all types of linear systems (consistent or inconsistent, full rank or rank-deficient). In this paper, we interpret the randomized extended Kaczmarz and Gauss-Seidel algorithms as specific combinations of the randomized Kaczmarz and Gauss-Seidel algorithms and present refined upper bounds for their convergence.
We propose a simple doubly stochastic block Gauss--Seidel algorithm for solving linear systems of equations. By varying the row partition parameter and the column partition parameter of the coefficient matrix, we recover the Landweber algorithm, the randomized Kaczmarz algorithm, the randomized Gauss--Seidel algorithm, and the doubly stochastic Gauss--Seidel algorithm. For general (consistent or inconsistent) linear systems, we show the exponential convergence of the {\it norms of the expected iterates} via exact formulas. For consistent linear systems, we prove the exponential convergence of …
We present a novel greedy Gauss-Seidel method for solving large linear least squares problem. This method improves the greedy randomized coordinate descent (GRCD) method proposed recently by Bai and Wu [Bai ZZ, and Wu WT. On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer Linear Algebra Appl. 2019;26(4):1--15], which in turn improves the popular randomized Gauss-Seidel method. Convergence analysis of the new method is provided. Numerical experiments show that, for the same accuracy, our method outperforms the GRCD method in term of the comput…
The Kaczmarz algorithm is one of the most popular methods for solving large-scale over-determined linear systems due to its simplicity and computational efficiency. This method can be viewed as a special instance of a more general class of sketch and project methods. Recently, a block Gaussian version was proposed that uses a block Gaussian sketch, enjoying the regularization properties of Gaussian sketching, combined with the acceleration of the block variants. Theoretical analysis was only provided for the non-block version of the Gaussian sketch method. Here, we provide theoretical guarante…
In this paper, a Gauss-Seidel method with oblique direction (GSO) is proposed for finding the least-squares solution to a system of linear equations, where the coefficient matrix may be full rank or rank deficient and the system is overdetermined or underdetermined. Through this method, the number of iteration steps and running time can be reduced to a greater extent to find the least-squares solution, especially when the columns of matrix A are close to linear correlation. It is theoretically proved that GSO method converges to the least-squares solution. At the same time, a randomized versio…
With a greedy strategy to construct control index set of coordinates firstly and then choosing the corresponding column submatrix in each iteration, we present a greedy block Gauss-Seidel (GBGS) method for solving large linear least squares problem. Theoretical analysis demonstrates that the convergence factor of the GBGS method can be much smaller than that of the greedy randomized coordinate descent (GRCD) method proposed recently in the literature. On the basis of the GBGS method, we further present a pseudoinverse-free greedy block Gauss-Seidel method, which doesn't need to calculate the M…
Several applications in optimization, image, and signal processing deal with data that belong to the Stiefel manifold St(n,p), that is, the set of n-by-p matrices with orthonormal columns. Some applications, like the Riemannian center of mass, require evaluating the Riemannian distance between two arbitrary points on St(n,p). This can be done by explicitly constructing the geodesic connecting these two points. An existing method for finding geodesics is the leapfrog algorithm of J. L. Noakes. This algorithm is related to the Gauss-Seidel method, a classical iterative method for solving a linea…
Randomized iterative algorithms have attracted much attention in recent years because they can approximately solve large-scale linear systems of equations without accessing the entire coefficient matrix. In this paper, we propose two novel pseudoinverse-free randomized block iterative algorithms for solving consistent and inconsistent linear systems. The proposed algorithms require two user-defined random matrices: one for row sampling and the other for column sampling. We can recover the well-known doubly stochastic Gauss--Seidel, randomized Kaczmarz, randomized coordinate descent, and random…
The randomized Gauss--Seidel method and its extension have attracted much attention recently and their convergence rates have been considered extensively. However, the convergence rates are usually determined by upper bounds, which cannot fully reflect the actual convergence. In this paper, we make a detailed analysis of their convergence behaviors. The analysis shows that the larger the singular value of $A$ is, the faster the error decays in the corresponding singular vector space, and the convergence directions are mainly driven by the large singular values at the beginning, then gradually …
In this paper, we present two improved Gauss-Seidel projection methods with unconditional stability. The first method updates the gyromagnetic term and the damping term simultaneously and follows by a projection step. The second method introduces two sets of approximate solutions, where we update the gyromagnetic term and the damping term simultaneously for one set of approximate solutions and apply the projection step to the other set of approximate solutions in an alternating manner. Compared to the original Gauss-Seidel projection method which has to solve heat equations $7$ times at each t…
We consider linear systems $Ax = b$ where $A \in \mathbb{R}^{m \times n}$ consists of normalized rows, $\|a_i\|_{\ell^2} = 1$, and where up to $βm$ entries of $b$ have been corrupted (possibly by arbitrarily large numbers). Haddock, Needell, Rebrova and Swartworth propose a quantile-based Random Kaczmarz method and show that for certain random matrices $A$ it converges with high likelihood to the true solution. We prove a deterministic version by constructing, for any matrix $A$, a number $β_A$ such that there is convergence for all perturbations with $β< β_A$. Assuming a random matrix heurist…
Aitken extrapolation normally applied to convergent fixed point iteration is extended to extrapolate the solution of a divergent iteration. In addition, higher order Aitken extrapolation is introduced that enables successive decomposition of high Eigen values of the iteration matrix to enable convergence. While extrapolation of a convergent fixed point iteration using a geometric series sum is a known form of Aitken acceleration, it is shown in this paper that the same formula can be used to estimate the solution of sets of linear equations from diverging Gauss Seidel iterations. In both conve…
A randomized Kaczmarz method was recently proposed for phase retrieval, which has been shown numerically to exhibit empirical performance over other state-of-the-art phase retrieval algorithms both in terms of the sampling complexity and in terms of computation time. While the rate of convergence has been studied well in the real case where the signals and measurement vectors are all real-valued, there is no guarantee for the convergence in the complex case. In fact, the linear convergence of the randomized Kaczmarz method for phase retrieval in the complex setting is left as a conjecture by T…
The GMRES algorithm of Saad and Schultz (1986) is an iterative method for approximately solving linear systems $A{\bf x}={\bf b}$, with initial guess ${\bf x}_0$ and residual ${\bf r}_0 = {\bf b} - A{\bf x}_0$. The algorithm employs the Arnoldi process to generate the Krylov basis vectors (the columns of $V_k$). It is well known that this process can be viewed as a $QR$ factorization of the matrix $B_k = [\: {\bf r}_0, AV_k\:]$ at each iteration. Despite an ${O}(ε)κ(B_k)$ loss of orthogonality, for unit roundoff $ε$ and condition number $κ$, the modified Gram-Schmidt formulation was shown to b…
Large-scale linear complementarity problems (LCPs) are repeatedly solved in interactive rigid-body simulations. The projected Gauss-Seidel method is often employed for LCPs, since it has advantages in computation time, numerical robustness, and memory use. Zheng and Yin (2013) proposed modulus-based matrix splitting iteration methods and showed their effectiveness for large problems, but a simple application of their approach to large-scale LCPs in interactive rigid-body simulations is not effective since such a simple application demands large matrix multiplications. In this paper, we propose…
Exploiting Task-Based Parallelism for the Red-Black Gauss-Seidel Method on 2D Grids
KTH Royal Institute of Technology（皇家理工学院）；Forschungszentrum Jülich GmbH（于利希研究中心有限公司）；Bergische Universität Wuppertal（伍珀塔尔大学）；University of Groningen（格罗宁根大学）
Some variants of the (block) Gauss--Seidel iteration for the solution of linear systems with $M$-matrices in (block) Hessenberg form are discussed. Comparison results for the asymptotic convergence rate of some regular splittings are derived: in particular, we prove that for a lower-Hessenberg M-matrix $ρ(P_{GS})\geq ρ(P_S)\geq ρ(P_{AGS})$, where $P_{GS}, P_S, P_{AGS}$ are the iteration matrices of the Gauss--Seidel, staircase, and anti-Gauss--Seidel method. This is a result that does not seem to follow from classical comparison results, as these splittings are not directly comparable. It is s…
We extend results known for the randomized Gauss-Seidel and the Gauss-Southwell methods for the case of a Hermitian and positive definite matrix to certain classes of non-Hermitian matrices. We obtain convergence results for a whole range of parameters describing the probabilities in the randomized method or the greedy choice strategy in the Gauss-Southwell-type methods. We identify those choices which make our convergence bounds best possible. Our main tool is to use weighted l1-norms to measure the residuals. A major result is that the best convergence bounds that we obtain for the expected …
The fully implicit method is the most commonly used approach to solve black-oil problems in reservoir simulation. The method requires repeated linearization of large nonlinear systems and produces ill-condi\-tioned linear systems. We present a strategy to reduce computational time that relies on two key ideas: (\textit{i}) a sequential formulation that decouples flow and transport into separate subproblems, and (\textit{ii}) a highly efficient Gauss--Seidel solver for the transport problems. This solver uses intercell fluxes to reorder the grid cells according to their upstream neighbors, and …
Gauss-Seidel (GS) relaxation is often employed as a preconditioner for a Krylov solver or as a smoother for Algebraic Multigrid (AMG). However, the requisite sparse triangular solve is difficult to parallelize on many-core architectures such as graphics processing units (GPUs). In the present study, the performance of the traditional GS relaxation based on a triangular solve is compared with two-stage variants, replacing the direct triangular solve with a fixed number of inner Jacobi-Richardson (JR) iterations. When a small number of inner iterations is sufficient to maintain the Krylov conver…
Magnetization dynamics in magnetic materials is often modeled by the Landau-Lifshitz equation, which is solved numerically in general. In micromagnetic simulations, the computational cost relies heavily on the time-marching scheme and the evaluation of stray field. Explicit marching schemes are efficient but suffer from severe stability constraints, while nonlinear systems of equations have to be solved in implicit schemes though they are unconditionally stable. A better compromise between stability and efficiency is the semi-implicit scheme, such as the Gauss-Seidel projection method (GSPM) a…
Analysis of Block Jacobi/Gauss-Seidel and additive/multiplicative Schwarz preconditioning through the theory of GLT sequences, with applications to domain decomposition discretizations
University of Rome Tor Vergata（罗马托尔维加塔大学）；University Mohammed VI Polytechnic（穆罕默德六世polytechnic大学）；University of Insubria（因苏布里亚大学）；Uppsala University（乌普萨拉大学）
A Dual Symmetric Gauss-Seidel Alternating Direction Method of Multipliers for Hyperspectral Sparse Unmixing
School of Information Science and technology, and the Provincial Key Lab of Information Coding and Trans- mission, Southwest Jiaotong University（信息科学与技术学院，信息编码与传输省重点实验室，西南交通大学）；School of Mathematics, Southwest Jiaotong University（数学学院，西南交通大学）；School of Computer and Computing Science, Zhejiang University City College（计算机与计算科学学院，浙江大学城市学院）
