One of the greatest success stories of randomized algorithms for linear algebra has been the development of fast, randomized algorithms for highly overdetermined linear least-squares problems. However, none of the existing algorithms is backward stable, preventing them from being deployed as drop-in replacements for existing QR-based solvers. This paper introduces sketch-and-precondition with iterative refinement (SPIR) and FOSSILS, two provably backward stable randomized least-squares solvers. SPIR and FOSSILS combine iterative refinement with a preconditioned iterative method applied to the normal equations and converge at the same rate as existing randomized least-squares solvers. This work offers the promise of incorporating randomized least-squares solvers into existing software libraries while maintaining the same level of accuracy and stability as classical solvers.
Deterministic and randomized, row-action and column-action linear solvers have become increasingly popular owing to their simplicity, low computational and memory complexities, and ease of composition with other techniques. Moreover, in order to achieve high-performance, such solvers must often be adapted to the given problem structure and to the hardware platform on which the problem will be solved. Unfortunately, determining whether such adapted solvers will converge to a solution has required equally unique analyses. As a result, adapted, reliable solvers are slow to be developed and deploy…
Randomized orthogonal projection methods (ROPMs) can be used to speed up the computation of Krylov subspace methods in various contexts. Through a theoretical and numerical investigation, we establish that these methods produce quasi-optimal approximations over the Krylov subspace. Our numerical experiments outline the convergence of ROPMs for all matrices in our test set, with occasional spikes, but overall with a convergence rate similar to that of standard OPMs.
Randomized linear system solvers have become popular as they have the potential to reduce floating point complexity while still achieving desirable convergence rates. One particularly promising class of methods, random sketching solvers, has achieved the best known computational complexity bounds in theory, but is blunted by two practical considerations: there is no clear way of choosing the size of the sketching matrix apriori; and there is a nontrivial storage cost of the sketched system. In this work, we make progress towards addressing these issues by implicitly generating the sketched sys…
Regularization Properties of the Krylov Iterative Solvers CGME and LSMR For Linear Discrete Ill-Posed Problems with an Application to Truncated Randomized SVDs
Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching
University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
In this work, we develop efficient solvers for linear inverse problems based on randomized singular value decomposition (RSVD). This is achieved by combining RSVD with classical regularization methods, e.g., truncated singular value decomposition, Tikhonov regularization, and general Tikhonov regularization with a smoothness penalty. One distinct feature of the proposed approach is that it explicitly preserves the structure of the regularized solution in the sense that it always lies in the range of a certain adjoint operator. We provide error estimates between the approximation and the exact …
This survey describes probabilistic algorithms for linear algebra computations, such as factorizing matrices and solving linear systems. It focuses on techniques that have a proven track record for real-world problem instances. The paper treats both the theoretical foundations of the subject and the practical computational issues. Topics covered include norm estimation; matrix approximation by sampling; structured and unstructured random embeddings; linear regression problems; low-rank approximation; subspace iteration and Krylov methods; error estimation and adaptivity; interpolatory and CUR …
This paper describes a set of rational filtering algorithms to compute a few eigenvalues (and associated eigenvectors) of non-Hermitian matrix pencils. Our interest lies in computing eigenvalues located inside a given disk, and the proposed algorithms approximate these eigenvalues and associated eigenvectors by harmonic Rayleigh-Ritz projections on subspaces built by computing range spaces of rational matrix functions through randomized range finders. These rational matrix functions are designed so that directions associated with non-sought eigenvalues are dampened to (approximately) zero. Var…
Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra
University of Pavia（帕维亚大学）；Department of Mathematics（数学系）；Department of Civil Engineering and Architecture（土木工程与建筑系）；Institut Camille Jordan, Lyon 1 Université（让·乔丹研究所，里昂1大学）
This paper presents an adaptive randomized algorithm for computing the butterfly factorization of a $m\times n$ matrix with $m\approx n$ provided that both the matrix and its transpose can be rapidly applied to arbitrary vectors. The resulting factorization is composed of $O(\log n)$ sparse factors, each containing $O(n)$ nonzero entries. The factorization can be attained using $O(n^{3/2}\log n)$ computation and $O(n\log n)$ memory resources. The proposed algorithm applies to matrices with strong and weak admissibility conditions arising from surface integral equation solvers with a rigorous e…
The randomized projection (RP) method is a simple iterative scheme for solving linear feasibility problems and has recently gained popularity due to its speed and low memory requirement. This paper develops an accelerated variant of the standard RP method by using two ingredients: the greedy probability criterion and the average block approach, and obtains a greedy randomized average block projection (GRABP) method for solving large-scale systems of linear inequalities. We prove that this method converges linearly in expectation under different choices of extrapolated stepsizes. Numerical expe…
Multilevel methods are among the most efficient numerical methods for solving large-scale linear systems that arise from discretized partial differential equations. The fundamental module of such methods is a two-level procedure, which consists of compatible relaxation and coarse-level correction. Regarding two-level convergence theory, most previous works focus on the case of exact (Galerkin) coarse solver. In practice, however, it is often too costly to solve the Galerkin coarse-level system exactly when its size is relatively large. Compared with the exact case, the convergence theory of in…
In applications like medical imaging, error correction, and sensor networks, one needs to solve large-scale linear systems that may be corrupted by a small number of arbitrarily large corruptions. We consider solving such large-scale systems of linear equations $A\mathbf{x}=\mathbf{b}$ that are inconsistent due to corruptions in the measurement vector $\mathbf{b}$. With this as our motivating example, we develop an approach for this setting that allows detection of the corrupted entries and thus convergence to the "true" solution of the original system. We provide analytical justification for …
A Butterfly-Based Direct Integral Equation Solver Using Hierarchical LU Factorization for Analyzing Scattering from Electrically Large Conducting Objects
Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气与计算机工程系）；University of Michigan（密歇根大学）；Department of Microwave Engineering, University of Electronic Science and Technology of China（电子科技大学微波工程系）；University of Electronic Science and Technology of China（电子科技大学）
Pauli-structured preconditioning for quantum linear system solvers
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）
Algorithm librla: A library of randomized linear algebra routines
Department of Applied Mathematics, University of Colorado Boulder（科罗拉多大学博尔德分校应用数学系）；National Institute of Standards and Technology (NIST)（美国国家标准与技术研究院）
Randomized Row and Column Iterative Methods with a Quantum Computer
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematics and Statistics, Wuhan University（武汉大学数学与统计学院）
With the growth of large data as well as large-scale learning tasks, the need for efficient and robust linear system solvers is greater than ever. The randomized Kaczmarz method (RK) and similar stochastic iterative methods have received considerable recent attention due to their efficient implementation and memory footprint. These methods can tolerate streaming data, accessing only part of the data at a time, and can also approximate the least squares solution even if the system is affected by noise. However, when data is instead affected by large (possibly adversarial) corruptions, these met…
A Note on Randomized Kaczmarz Algorithm for Solving Doubly-Noisy Linear Systems
College of Computing, Mohammed VI Polytechnic University（穆罕默德六世理工大学计算机学院）；University of Central Florida（中佛罗里达大学）；University of California, Irvine（加州大学欧文分校）
Given a matrix $A$ of size $m\times n$, the manuscript describes a algorithm for computing a QR factorization $AP=QR$ where $P$ is a permutation matrix, $Q$ is orthonormal, and $R$ is upper triangular. The algorithm is blocked, to allow it to be implemented efficiently. The need for single vector pivoting in classical algorithms for computing QR factorizations is avoided by the use of randomized sampling to find blocks of pivot vectors at once. The advantage of blocking becomes particularly pronounced when $A$ is very large, and possibly stored out-of-core, or on a distributed memory machine. …
Quantile Randomized Kaczmarz for Streaming Linear Systems with Massart Noise
University of California, Irvine（加州大学尔湾分校）；Hong Kong University of Science and Technology（香港科技大学）；Columbia University（哥伦比亚大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Numerical solutions to the Eikonal equation are computed using variants of the fast marching method, the fast sweeping method, and the fast iterative method. In this paper, we provide a unified view of these algorithms that highlights their similarities and suggests a wider class of Eikonal solvers. We then use this framework to justify applying concurrent priority scheduling techniques to Eikonal solvers. We demonstrate that doing so results in good parallel performance for a problem from seismology. We explain why existing Eikonal solvers may produce different results despite using the same …
