We consider least-squares problems with quadratic regularization and propose novel sketching-based iterative methods with an adaptive sketch size. The sketch size can be as small as the effective dimension of the data matrix to guarantee linear convergence. However, a major difficulty in choosing the sketch size in terms of the effective dimension lies in the fact that the latter is usually unknown in practice. Current sketching-based solvers for regularized least-squares fall short on addressing this issue. Our main contribution is to propose adaptive versions of standard sketching-based iterative solvers, namely, the iterative Hessian sketch and the preconditioned conjugate gradient method, that do not require a priori estimation of the effective dimension. We propose an adaptive mechanism to control the sketch size according to the progress made in each step of the iterative solver. If enough progress is not made, the sketch size increases to improve the convergence rate. We prove that the adaptive sketch size scales at most in terms of the effective dimension, and that our adaptive methods are guaranteed to converge linearly. Consequently, our adaptive methods improve the state-of-the-art complexity for solving dense, ill-conditioned least-squares problems. Importantly, we illustrate numerically on several synthetic and real datasets that our method is extremely efficient and is often significantly faster than standard least-squares solvers such as a direct factorization based solver, the conjugate gradient method and its preconditioned variants.
Solving systems of linear equations is a problem occuring frequently in water engineering applications. Usually the size of the problem is too large to be solved via direct factorization. One can resort to iterative approaches, in particular the conjugate gradients method if the matrix is symmetric positive definite. Preconditioners further enhance the rate of convergence but hitherto only handcrafted ones requiring expert knowledge have been used. We propose an innovative approach employing Machine Learning, in particular a Convolutional Neural Network, to unassistedly design preconditioning …
Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching
University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
University of Oxford（牛津大学）；European Centre For Medium Range Weather Forecasts（欧洲中期天气预报中心）；National Center for Atmospheric Research（美国国家大气研究中心）
It is tested whether machine learning methods can be used for preconditioning to increase the performance of the linear solver -- the backbone of the semi-implicit, grid-point model approach for weather and climate models. Embedding the machine-learning method within the framework of a linear solver circumvents potential robustness issues that machine learning approaches are often criticized for, as the linear solver ensures that a sufficient, pre-set level of accuracy is reached. The approach does not require prior availability of a conventional preconditioner and is highly flexible regarding…
In this paper, we present a data-driven approach to iteratively solve the discrete heterogeneous Helmholtz equation at high wavenumbers. In our approach, we combine classical iterative solvers with convolutional neural networks (CNNs) to form a preconditioner which is applied within a Krylov solver. For the preconditioner, we use a CNN of type U-Net that operates in conjunction with multigrid ingredients. Two types of preconditioners are proposed 1) U-Net as a coarse grid solver, and 2) U-Net as a deflation operator with shifted Laplacian V-cycles. Following our training scheme and data-augmen…
NeuraLSP: A Neural Spectral Preconditioner for Accelerating PDE Solvers
Department of Applied Mathematics and Statistics, Stony Brook University（应用数学与统计学系，史泰尼斯布鲁克大学）；Department of Computer Science, Stony Brook University（计算机科学系，史泰尼斯布鲁克大学）
Learning from Linear Algebra: A Graph Neural Network Approach to Preconditioner Design for Conjugate Gradient Solvers
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Sberbank of Russia（俄罗斯联邦储蓄银行）；Artificial Intelligence Research Institute (AIRI)（人工智能研究所（AIRI））；Fraunhofer Institute for Industrial Mathematics ITWM（弗劳恩霍夫工业数学研究所 ITWM）；Institute of Computational Mathematics and Mathematical Geophysics SB RAS（俄罗斯科学院西伯利亚分院计算数学与数学地球物理研究所）
BandPC: Learning Residual-Band Preconditioner Combinations for Flexible Conjugate Gradient Solvers
Wuhan University of Technology（武汉理工大学）；Embodied Physical Intelligence Center in Civil Engineering, Wuhan University of Technology（武汉理工大学土木工程具身智能中心）；Key Laboratory of Low-Altitude Technology and Smart Urban Renewal, Department of Housing and Urban-Rural Development of Hubei Province, Wuhan University of Technology（湖北省住房和城乡建设厅低空技术与智慧城市更新重点实验室（武汉理工大学））；Sanya Science and Education Innovation Park of Wuhan University of Technology（武汉理工大学三亚科教创新园）
Iterative Hessian sketch (IHS) is an effective sketching method for modeling large-scale data. It was originally proposed by Pilanci and Wainwright (2016; JMLR) based on randomized sketching matrices. However, it is computationally intensive due to the iterative sketch process. In this paper, we analyze the IHS algorithm under the unconstrained least squares problem setting, then propose a deterministic approach for improving IHS via A-optimal subsampling. Our contributions are three-fold: (1) a good initial estimator based on the A-optimal design is suggested; (2) a novel ridged preconditione…
We propose a new randomized algorithm for solving L2-regularized least-squares problems based on sketching. We consider two of the most popular random embeddings, namely, Gaussian embeddings and the Subsampled Randomized Hadamard Transform (SRHT). While current randomized solvers for least-squares optimization prescribe an embedding dimension at least greater than the data dimension, we show that the embedding dimension can be reduced to the effective dimension of the optimization problem, and still preserve high-probability convergence guarantees. In this regard, we derive sharp matrix deviat…
Prediction of Multiscale Features Using Deep Learning-based Preconditioner-Solver Architecture for Darcy Equation in High-Contrast Media
School of Mathematics and Physics, Xi'an Jiaotong-Liverpool University（数学与物理学院，西安交通大学利物浦大学）；Department of Mathematical Sciences, University of Liverpool（数学科学系，利物浦大学）；School of Mathematics and Information Science, Yantai University（数学与信息科学学院，烟台大学）
Multi-Level GNN Preconditioner for Solving Large Scale Problems
Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Direction Technologie et Numerique IFP Énergies-Nouvelles（IFP新能源公司技术与数字部）；Safran Tech（赛峰科技）
