在本报告中，我们提出了一种通用且高效的预处理安德森加速（preconditioned Anderson acceleration，PAA）方法，用于不动点迭代。该框架在收敛速度（线性、超线性或二次）与雅可比矩阵相关的计算成本之间提供了灵活的平衡。我们的方法涵盖了多种不动点迭代技术，包括Picard迭代、Newton迭代和拟Newton迭代。PAA方法可被理解为将安德森加速（Anderson acceleration，AA）用作自身的预处理器，或是在拟Newton方法收敛不足时作为其加速器。该方法可适配具有不同非线性程度与复杂度的各类问题，通过引入合适的预处理器提升了收敛速度与鲁棒性。我们在多种问题上测试了多种预处理策略，并研究了预处理器的延迟更新策略以进一步降低计算成本。
英文摘要：
In this report, we present a versatile and efficient preconditioned Anderson acceleration (PAA) method for fixed-point iterations. The proposed framework offers flexibility in balancing convergence rates (linear, super-linear, or quadratic) and computational costs related to the Jacobian matrix. Our approach recovers various fixed-point iteration techniques, including Picard, Newton, and quasi-Newton iterations. The PAA method can be interpreted as employing Anderson acceleration (AA) as its own preconditioner or as an accelerator for quasi-Newton methods when their convergence is insufficient. Adaptable to a wide range of problems with differing degrees of nonlinearity and complexity, the method achieves improved convergence rates and robustness by incorporating suitable preconditioners. We test multiple preconditioning strategies on various problems and investigate a delayed update strategy for preconditioners to further reduce the computational costs.
Efficient Solution of Generalized Sylvester Equations via Preconditioned Alternating Anderson Acceleration
Nanchang University（南昌大学）；Nanchang Hangkong University（南昌航空大学）；Fujian Agriculture and Forestry University（福建农林大学）；Dalian University of Technology（大连理工大学）
This paper examines a number of extrapolation and acceleration methods, and introduces a few modifications of the standard Shanks transformation that deal with general sequences. One of the goals of the paper is to lay out a general framework that encompasses most of the known acceleration strategies. The paper also considers the Anderson Acceleration method under a new light and exploits a connection with quasi-Newton methods, in order to establish local linear convergence results of a stabilized version of Anderson Acceleration method. The methods are tested on a number of problems, includin…
In this paper, we focus on solving a sequence of linear systems with an identical (or similar) coefficient matrix. For this type of problems, we investigate the subspace correction and deflation methods, which use an auxiliary matrix (subspace) to accelerate the convergence of the iterative method. In practical simulations, these acceleration methods typically work well when the range of the auxiliary matrix contains eigenspaces corresponding to small eigenvalues of the coefficient matrix. We have developed a new algebraic auxiliary matrix construction method based on error vector sampling, in…
Anderson acceleration (AA) has a long history of use and a strong recent interest due to its potential ability to dramatically improve the linear convergence of the fixed-point iteration. Most authors are simply using and analyzing the stationary version of Anderson acceleration (sAA) with a constant damping factor or without damping. Little attention has been paid to nonstationary algorithms. However, damping can be useful and is sometimes crucial for simulations in which the underlying fixed-point operator is not globally contractive. The role of this damping factor has not been fully unders…
In this paper, we propose and analyze a set of fully non-stationary Anderson acceleration algorithms with dynamic window sizes and optimized damping. Although Anderson acceleration (AA) has been used for decades to speed up nonlinear solvers in many applications, most authors are simply using and analyzing the stationary version of Anderson acceleration (sAA) with fixed window size and a constant damping factor. The behavior and potential of the non-stationary version of Anderson acceleration methods remain an open question. Since most efficient linear solvers use composable algorithmic compon…
We consider two modifications of the Arrow-Hurwicz (AH) iteration for solving the incompressible steady Navier-Stokes equations for the purpose of accelerating the algorithm: grad-div stabilization, and Anderson acceleration. AH is a classical iteration for general saddle point linear systems and it was later extended to Navier-Stokes iterations in the 1970's which has recently come under study again. We apply recently developed ideas for grad-div stabilization and divergence-free finite element methods along with Anderson acceleration of fixed point iterations to AH in order to improve its co…
We prove non asymptotic linear convergence rates for the constrained Anderson acceleration extrapolation scheme. These guarantees come from new upper bounds on the constrained Chebyshev problem, which consists in minimizing the maximum absolute value of a polynomial on a bounded real interval with $l_1$ constraints on its coefficients vector. Constrained Anderson Acceleration has a numerical cost comparable to that of the original scheme.
Anderson acceleration (AA) is a technique for accelerating the convergence of fixed-point iterations. In this paper, we apply AA to a sequence of functions and modify the norm in its internal optimization problem to the $\mathcal{H}^{-s}$ norm, for some positive integer $s$, to bias it towards low-frequency spectral content in the residual. We analyze the convergence of AA by quantifying its improvement over Picard iteration. We find that AA based on the $\mathcal{H}^{-2}$ norm is well-suited to solve fixed-point operators derived from second-order elliptic differential operators, including th…
This paper develops an efficient and robust solution technique for the steady Boussinesq model of non-isothermal flow using Anderson acceleration applied to a Picard iteration. After analyzing the fixed point operator associated with the nonlinear iteration to prove that certain stability and regularity properties hold, we apply the authors' recently constructed theory for Anderson acceleration, which yields a convergence result for the Anderson accelerated Picard iteration for the Boussinesq system. The result shows that the leading term in the residual is improved by the gain in the optimiza…
This paper studies a finite element discretization of the regularized Bingham equations that describe viscoplastic flow. An efficient nonlinear solver for the discrete model is then proposed and analyzed. The solver is based on Anderson acceleration (AA) applied to a Picard iteration, and we show accelerated convergence of the method by applying AA theory (recently developed by the authors) to the iteration, after showing sufficient smoothness properties of the associated fixed point operator. Numerical tests of spatial convergence are provided, as are results of the model for 2D and 3D driven…
This paper studies the performance Newton's iteration applied with Anderson acceleration for solving the incompressible steady Navier-Stokes equations. We manifest that this method converges superlinearly with a good initial guess, and moreover, a large Anderson depth decelerates the convergence speed comparing to a small Anderson depth. We observe that the numerical tests confirm these analytical convergence results, and in addition, Anderson acceleration sometimes enlarges the domain of convergence for Newton's method.
This paper considers an enhancement of the classical iterated penalty Picard (IPP) method for the incompressible Navier-Stokes equations, where we restrict our attention to $O(1)$ penalty parameter, and Anderson acceleration (AA) is used to significantly improve its convergence properties. After showing the fixed point operator associated with the IPP iteration is Lipschitz continuous and Lipschitz continuously (Frechet) differentiable, we apply a recently developed general theory for AA to conclude that IPP enhanced with AA improves its linear convergence rate by the gain factor associated wi…
This work introduces, analyzes and demonstrates an efficient and theoretically sound filtering strategy to ensure the condition of the least-squares problem solved at each iteration of Anderson acceleration. The filtering strategy consists of two steps: the first controls the length disparity between columns of the least-squares matrix, and the second enforces a lower bound on the angles between subspaces spanned by the columns of that matrix. The combined strategy is shown to control the condition number of the least-squares matrix at each iteration. The method is shown to be effective on a r…
A one-step analysis of Anderson acceleration with general algorithmic depths is presented. The resulting residual bounds within both contractive and noncontractive settings reveal the balance between the contributions from the higher and lower order terms, which are both dependent on the success of the optimization problem solved at each step of the algorithm. The new residual bounds show the additional terms introduced by the extrapolation produce terms that are of a higher order than was previously understood. In the contractive setting, these bounds sharpen previous convergence and accelera…
Two-Level Sketching Alternating Anderson acceleration for Complex Physics Applications
Imagination Corp.（Imagination公司）；Department of Applied Mathematics, Fictional University（虚构大学应用数学系）；Instituto de Ingeniería Matemática y Computacional & Facultad de Ciencias Biológicas, Pontificia Universidad Católica de Chile（智利天主教大学数学与计算工程研究所及生物科学学院）；Centro de Modelamiento Matemático（数学建模中心）；Computational Sciences and Engineering Division, Oak Ridge National Laboratory（橡树岭国家实验室计算科学与工程部）
The Dynamical Anatomy of Anderson Acceleration:From Adaptive Momentum to Variable-Mass ODEs
College of Mathematics and Statistics, Nanjing University of Information Science and Technology（南京信息工程大学数学与统计学院）；Delft Institute of Applied Mathematics, Delft University of Technology（代尔夫特理工大学应用数学研究所）
A proof that Anderson acceleration improves the convergence rate in linearly converging fixed point methods (but not in those converging quadratically)
Clemson University（克莱姆森大学）；University of Florida（佛罗里达大学）；College of William & Mary（威廉玛丽学院）
Emory University（埃默里大学）；University of Minnesota（明尼苏达大学）
Many modern machine learning algorithms such as generative adversarial networks (GANs) and adversarial training can be formulated as minimax optimization. Gradient descent ascent (GDA) is the most commonly used algorithm due to its simplicity. However, GDA can converge to non-optimal minimax points. We propose a new minimax optimization framework, GDA-AM, that views the GDAdynamics as a fixed-point iteration and solves it using Anderson Mixing to con-verge to the local minimax. It addresses the diverging issue of simultaneous GDAand accelerates the convergence of alternating GDA. We show theor…
