In this paper, we propose, analyze and demonstrate a dynamic momentum method to accelerate power and inverse power iterations with minimal computational overhead. The method can be applied to real diagonalizable matrices, is provably convergent with acceleration in the symmetric case, and does not require a priori spectral knowledge. We review and extend background results on previously developed static momentum accelerations for the power iteration through the connection between the momentum accelerated iteration and the standard power iteration applied to an augmented matrix. We show that the augmented matrix is defective for the optimal parameter choice. We then present our dynamic method which updates the momentum parameter at each iteration based on the Rayleigh quotient and two previous residuals. We present convergence and stability theory for the method by considering a power-like method consisting of multiplying an initial vector by a sequence of augmented matrices. We demonstrate the developed method on a number of benchmark problems, and see that it outperforms both the power iteration and often the static momentum acceleration with optimal parameter choice. Finally, we present and demonstrate an explicit extension of the algorithm to inverse power iterations.
Accelerating operator Sinkhorn iteration with overrelaxation
The Institute of Statistical Mathematics(东京统计数学研究所);Institute of Mathematics & Centre for Advanced Analytics and Predictive Sciences, University of Augsburg(奥格斯堡大学数学研究所与高级分析与预测科学中心)
Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs
University of Utah(犹他大学);University of California, Los Angeles(加州大学洛杉矶分校);Illinois Institute of Technology(伊利诺伊理工学院);University of California, Irvine(加州大学欧文分校);Scientific Computing and Imaging (SCI) Institute(科学计算与成像研究所)
Accelerating AI Performance using Anderson Extrapolation on GPUs
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Applied Physics Program, Physical Sciences and Engineering Division(物理科学与工程分部应用物理项目);Extreme Computing Research Center, Computer, Electrical and Mathematical Sciences and Engineering Division(计算机、电气与数学科学与工程分部极端计算研究中心)