A data scalable augmented Lagrangian KKT preconditioner for large scale inverse problems
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Institute for Computational Engineering and Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校计算工程与科学研究所）
Nick Alger, Umberto Villa, Tan Bui-Thanh, Omar Ghattas
备注：
30 pages, 4 figures, 1 table. Accepted for publication in SIAM Journal on Scientific Computing (SISC)
Current state of the art preconditioners for the reduced Hessian and the Karush-Kuhn-Tucker (KKT) operator for large scale inverse problems are typically based on approximating the reduced Hessian with the regularization operator. However, the quality of this approximation degrades with increasingly informative observations or data. Thus the best case scenario from a scientific standpoint (fully informative data) is the worse case scenario from a computational perspective. In this paper we present an augmented Lagrangian-type preconditioner based on a block diagonal approximation of the augmented upper left block of the KKT operator. The preconditioner requires solvers for two linear subproblems that arise in the augmented KKT operator, which we expect to be much easier to precondition than the reduced Hessian. Analysis of the spectrum of the preconditioned KKT operator indicates that the preconditioner is effective when the regularization is chosen appropriately. In particular, it is effective when the regularization does not over-penalize highly informed parameter modes and does not under-penalize uninformed modes. Finally, we present a numerical study for a large data/low noise Poisson source inversion problem, demonstrating the effectiveness of the preconditioner. In this example, three MINRES iterations on the KKT system with our preconditioner results in a reconstruction with better accuracy than 50 iterations of CG on the reduced Hessian system with regularization preconditioning.
Updating constraint preconditioners for KKT systems in quadratic programming via low-rank corrections
Università degli Studi di Firenze（佛罗伦萨大学）；Seconda Università degli Studi di Napoli（那不勒斯第二大学）；Istituto di Calcolo e Reti ad Alte Prestazioni, CNR（意大利国家研究委员会高性能计算与网络研究所）
We propose a novel preconditioned inexact primal-dual interior point method for constrained convex quadratic programming problems. The algorithm we describe invokes the preconditioned conjugate gradient method on a new reduced Schur complement KKT system, in implicit form. In contrast to standard approaches, the Schur complement formulation we consider enables reuse of the factorization of the KKT matrix with rows and columns corresponding to inequality constraints excluded, across all interior point iterations. Further, two new preconditioners are presented for the resulting reduced system, t…
The paper discusses a reuse of matrix factorization as a building block in the Augmented Lagrangian (AL) and modified AL preconditioners for non-symmetric saddle point linear algebraic systems. The strategy is applied to solve two-dimensional incompressible fluid problems with efficiency rates independent of the Reynolds number. The solver is then tested to simulate motion of a surface fluid, an example of a 2D flow motivated by an interest in lateral fluidity of inextensible viscous membranes. Numerical examples include the Kelvin--Helmholtz instability problem posed on the sphere and on the …
We propose an augmented Lagrangian preconditioner for a three-field stress-velocity-pressure discretization of stationary non-Newtonian incompressible flow with an implicit constitutive relation of power-law type. The discretization employed makes use of the divergence-free Scott-Vogelius pair for the velocity and pressure. The preconditioner builds on the work [P. E. Farrell, L. Mitchell, and F. Wechsung, SIAM J. Sci. Comput., 41 (2019), pp. A3073-A3096], where a Reynolds-robust preconditioner for the three-dimensional Newtonian system was introduced. The preconditioner employs a specialized …
We propose a robust and efficient augmented Lagrangian-type preconditioner for solving linearizations of the Oseen-Frank model arising in cholesteric liquid crystals. By applying the augmented Lagrangian method, the Schur complement of the director block can be better approximated by the weighted mass matrix of the Lagrange multiplier, at the cost of making the augmented director block harder to solve. In order to solve the augmented director block, we develop a robust multigrid algorithm which includes an additive Schwarz relaxation that captures a pointwise version of the kernel of the semi-…
In Benzi & Olshanskii (SIAM J.~Sci.~Comput., 28(6) (2006)) a preconditioner of augmented Lagrangian type was presented for the two-dimensional stationary incompressible Navier--Stokes equations that exhibits convergence almost independent of Reynolds number. The algorithm relies on a highly specialized multigrid method involving a custom prolongation operator and is tightly coupled to the use of piecewise constant finite elements for the pressure. However, the prolongation operator and velocity element used do not directly extend to three dimensions: the local solves necessary in the prolongat…
The magnetohydrodynamics (MHD) equations are generally known to be difficult to solve numerically, due to their highly nonlinear structure and the strong coupling between the electromagnetic and hydrodynamic variables, especially for high Reynolds and coupling numbers. In this work, we present a scalable augmented Lagrangian preconditioner for a finite element discretization of the $\mathbf{B}$-$\mathbf{E}$ formulation of the incompressible viscoresistive MHD equations. For stationary problems, our solver achieves robust performance with respect to the Reynolds and coupling numbers in two dime…
We devise 3-field and 4-field finite element approximations of a system describing the steady state of an incompressible heat-conducting fluid with implicit non-Newtonian rheology. We prove that the sequence of numerical approximations converges to a weak solution of the problem. We develop a block preconditioner based on augmented Lagrangian stabilisation for a discretisation based on the Scott-Vogelius finite element pair for the velocity and pressure. The preconditioner involves a specialised multigrid algorithm that makes use of a space-decomposition that captures the kernel of the diverge…
Topology optimization problems generally support multiple local minima, and real-world applications are typically three-dimensional. In previous work [I. P. A. Papadopoulos, P. E. Farrell, and T. M. Surowiec, Computing multiple solutions of topology optimization problems, SIAM Journal on Scientific Computing, (2021)], the authors developed the deflated barrier method, an algorithm that can systematically compute multiple solutions of topology optimization problems. In this work we develop preconditioners for the linear systems arising in the application of this method to Stokes flow, making it…
Augmented Lagrangian preconditioners have successfully yielded Reynolds-robust preconditioners for the stationary incompressible Navier-Stokes equations, but only for specific discretizations. The discretizations for which these preconditioners have been designed possess error estimates which depend on the Reynolds number, with the discretization error deteriorating as the Reynolds number is increased. In this paper we present an augmented Lagrangian preconditioner for the Scott-Vogelius discretization on barycentrically-refined meshes. This achieves both Reynolds-robust performance and Reynol…
In this study, we present an $hp$-multigrid preconditioner for a divergence-conforming HDG scheme for the generalized Stokes and the Navier-Stokes equations using an augmented Lagrangian formulation. Our method relies on conforming simplicial meshes in two- and three-dimensions. The $hp$-multigrid algorithm is a multiplicative auxiliary space preconditioner that employs the lowest-order space as the auxiliary space, and we developed a geometric multigrid method as the auxiliary space solver. For the generalized Stokes problem, the crucial ingredient of the geometric multigrid method is the equ…
In this paper, we present a data-driven approach to iteratively solve the discrete heterogeneous Helmholtz equation at high wavenumbers. In our approach, we combine classical iterative solvers with convolutional neural networks (CNNs) to form a preconditioner which is applied within a Krylov solver. For the preconditioner, we use a CNN of type U-Net that operates in conjunction with multigrid ingredients. Two types of preconditioners are proposed 1) U-Net as a coarse grid solver, and 2) U-Net as a deflation operator with shifted Laplacian V-cycles. Following our training scheme and data-augmen…
Novel Approach for solving the discrete Stokes problems based on Augmented Lagrangian and Global Techniques: Application to Saddle-Point Linear Systems from Incompressible flow
In this note we present a multigrid preconditioning method for solving quadratic optimization problems constrained by a fractional diffusion equation. Multigrid methods within the all-at-once approach to solve the first order-order optimality Karush-Kuhn-Tucker (KKT) systems are widely popular, but their development have relied on the underlying systems being sparse. On the other hand, for most discretizations, the matrix representation of fractional operators is expected to be dense. We develop a preconditioning strategy for our problem based on a reduced approach, namely we eliminate the sta…
In this paper we consider a class of convex conic programming. In particular, we first propose an inexact augmented Lagrangian (I-AL) method that resembles the classical I-AL method for solving this problem, in which the augmented Lagrangian subproblems are solved approximately by a variant of Nesterov's optimal first-order method. We show that the total number of first-order iterations of the proposed I-AL method for finding an $ε$-KKT solution is at most $\mathcal{O}(ε^{-7/4})$. We then propose an adaptively regularized I-AL method and show that it achieves a first-order iteration complexity…
In this paper we present general-purpose preconditioners for regularized augmented systems arising from optimization problems, and their corresponding normal equations. We discuss positive definite preconditioners, suitable for CG and MINRES. We consider "sparsifications" which avoid situations in which eigenvalues of the preconditioned matrix may become complex. Special attention is given to systems arising from the application of regularized interior point methods to linear or nonlinear convex programming problems.
We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact which allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of…
First-order methods (FOMs) have been widely used for solving large-scale problems. A majority of existing works focus on problems without constraint or with simple constraints. Several recent works have studied FOMs for problems with complicated functional constraints. In this paper, we design a novel augmented Lagrangian (AL) based FOM for solving problems with non-convex objective and convex constraint functions. The new method follows the framework of the proximal point (PP) method. On approximately solving PP subproblems, it mixes the usage of the inexact AL method (iALM) and the quadratic…
Augmented Lagrangian preconditioning for a simplified Ericksen--Leslie model of nematic liquid crystals
College of Science, National University of Defense Technology（国防科技大学理学院）；College of Meteorology and Oceanography, National University of Defense Technology（国防科技大学气象海洋学院）
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research（IBM研究院）；Michigan State University（密歇根州立大学）
Many real-world problems not only have complicated nonconvex functional constraints but also use a large number of data points. This motivates the design of efficient stochastic methods on finite-sum or expectation constrained problems. In this paper, we design and analyze stochastic inexact augmented Lagrangian methods (Stoc-iALM) to solve problems involving a nonconvex composite (i.e. smooth+nonsmooth) objective and nonconvex smooth functional constraints. We adopt the standard iALM framework and design a subroutine by using the momentum-based variance-reduced proximal stochastic gradient me…
We present augmented Lagrangian Schur complement preconditioners and robust multigrid methods for incompressible Stokes problems with extreme viscosity variations. Such Stokes systems arise, for instance, upon linearization of nonlinear viscous flow problems, and they can have severely inhomogeneous and anisotropic coefficients. Using an augmented Lagrangian formulation for the incompressibility constraint makes the Schur complement easier to approximate, but results in a nearly singular (1,1)-block in the Stokes system. We present eigenvalue estimates for the quality of the Schur complement a…
Two-level space-time domain decomposition methods for unsteady inverse problems
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Colorado Boulder（科罗拉多大学博尔德分校）；The Chinese University of Hong Kong（香港中文大学）
In recent years there has been a push to discover the governing equations dynamical systems directly from measurements of the state, often motivated by systems that are too complex to directly model. Although there has been substantial work put into such a discovery, doing so in the case of large noise has proved challenging. Here we develop an algorithm for Simultaneous Identification and Denoising of a Dynamical System (SIDDS). We infer the noise in the state measurements by requiring that the denoised data satisfies the dynamical system with an equality constraint. This is unlike existing w…
Accelerated Implicit GDA Schemes: Theoretical Guarantees and Application to Proximal Augmented Lagrangian Methods
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS)（上海数学与交叉学科研究院）；Research Institute of Intelligent Complex Systems, Fudan University（复旦大学智能复杂系统研究所）；Center for Mathematics and Interdisciplinary Sciences, Fudan University（复旦大学数学与交叉学科中心）
Subspace recycling techniques have been used quite successfully for the acceleration of iterative methods for solving large-scale linear systems. These methods often work by augmenting a solution subspace generated iteratively by a known algorithm with a fixed subspace of vectors which are ``useful'' for solving the problem. Often, this has the effect of inducing a projected version of the original linear system to which the known iterative method is then applied, and this projection can act as a deflation preconditioner, accelerating convergence. Most often, these methods have been applied fo…
