Nonlinear acceleration methods are powerful techniques to speed up fixed-point iterations. However, many acceleration methods require storing a large number of previous iterates and this can become impractical if computational resources are limited. In this paper, we propose a nonlinear Truncated Generalized Conjugate Residual method (nlTGCR) whose goal is to exploit the symmetry of the Hessian to reduce memory usage. The proposed method can be interpreted as either an inexact Newton or a quasi-Newton method. We show that, with the help of global strategies like residual check techniques, nlTGCR can converge globally for general nonlinear problems and that under mild conditions, nlTGCR is able to achieve superlinear convergence. We further analyze the convergence of nlTGCR in a stochastic setting. Numerical results demonstrate the superiority of nlTGCR when compared with several other competitive baseline approaches on a few problems. Our code will be available in the future.
Hessian-Free High-Resolution Nesterov Acceleration for Sampling
Georgia Institute of Technology(佐治亚理工学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院)
Graduate School of Artificial Intelligence(人工智能研究生院);Pohang University of Science and Technology(釜山科学技术大学);Department of Mathematics(数学系);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Konkuk University(康 kuk 大学)
Public Health School, Fujian Medical University(福建医科大学公共卫生学院);School of Mathematics and Statistics, Fuzhou University(福州大学数学与统计学院);Center for Applied Mathematics of Fujian Province(福建省应用数学中心);National Center for Applied Mathematics Shenzhen, SUSTech(南方科技大学深圳国家应用数学中心);Department of Mathematics, SUSTech(南方科技大学数学系);School of Science, BUPT(北京邮电大学理学院);The Smart City Research Institute of CETC(中国电子科技集团公司智慧城市研究院)