We develop and analyze a broad family of stochastic/randomized algorithms for inverting a matrix. We also develop specialized variants maintaining symmetry or positive definiteness of the iterates. All methods in the family converge globally and linearly (i.e., the error decays exponentially), with explicit rates. In special cases, we obtain stochastic block variants of several quasi-Newton updates, including bad Broyden (BB), good Broyden (GB), Powell-symmetric-Broyden (PSB), Davidon-Fletcher-Powell (DFP) and Broyden-Fletcher-Goldfarb-Shanno (BFGS). Ours are the first stochastic versions of these updates shown to converge to an inverse of a fixed matrix. Through a dual viewpoint we uncover a fundamental link between quasi-Newton updates and approximate inverse preconditioning. Further, we develop an adaptive variant of randomized block BFGS, where we modify the distribution underlying the stochasticity of the method throughout the iterative process to achieve faster convergence. By inverting several matrices from varied applications, we demonstrate that AdaRBFGS is highly competitive when compared to the well established Newton-Schulz and minimal residual methods. In particular, on large-scale problems our method outperforms the standard methods by orders of magnitude. Development of efficient methods for estimating the inverse of very large matrices is a much needed tool for preconditioning and variable metric optimization methods in the advent of the big data era.
A globally convergent modified Newton method for the direct minimization of the Ohta-Kawasaki energy with application to the directed self-assembly of diblock copolymers
Provably convergent Newton-Raphson methods for recovering primitive variables with applications to physical-constraint-preserving Hermite WENO schemes for relativistic hydrodynamics
Greedy Gauss-Newton algorithm for finding sparse solutions to nonlinear underdetermined systems of equations
School of Science and Technology, Örebro University（欧勒布鲁大学理学院与技术学院）；Department of Mathematical Sciences and Technology, Norwegian University of Life Sciences（挪威生命科学大学数学科学与技术系）
Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching
University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Online Covariance Matrix Estimation in Sketched Newton Methods
Department of Statistics, The University of Chicago（芝加哥大学统计系）；Mathematics and Computer Science Division, Argonne National Laboratory（阿贡国家实验室数学与计算机科学部）；School of Industrial and Systems Engineering, Georgia Institute of Technology（佐治亚理工学院工业与系统工程系）
