First-order operator splitting methods are ubiquitous among many fields through science and engineering, such as inverse problems, signal/image processing, statistics, data science and machine learning, to name a few. In this paper, we study a geometric property of first-order methods when applying to solve non-smooth optimization problems. With the tool of "partial smoothness", we design a framework to analyze the trajectory of the fixed-point sequence generated by first-order methods and show that locally, the fixed-point sequence settles onto a regular trajectory such as a straight line or a spiral. Based on this finding, we discuss the limitation of current widely used "inertial acceleration" technique, and propose a trajectory following adaptive acceleration algorithm. Global convergence is established for the proposed acceleration scheme based on the perturbation of fixed-point iteration. Locally, we first build connections between the acceleration scheme and the well-studied "vector extrapolation technique" in the field of numerical analysis, and then discuss local acceleration guarantees of the proposed acceleration scheme. Moreover, our result provides a geometric interpretation of these vector extrapolation techniques. Numerical experiments on various first-order methods are provided to demonstrate the advantage of the proposed adaptive acceleration scheme.
Online Adaptive Methods, Universality and Acceleration
ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）
我们提出了一种新的方法，凸无约束优化，没有任何修改，确保：（i）加速收敛速度光滑的目标，（ii）在一般（非光滑）设置的标准收敛速度，和（iii）在随机优化设置的标准收敛速度。据我们所知，这是第一个同时适用于所有上述设置的方法。我们方法的核心是自适应学习率规则，其采用重要性权重，本着自适应在线学习算法的精神（Duchi et al.，2011; Levy，2017），结合线性耦合两个序列的更新，本着（Allen-Zhu和Orecchia，2017）的精神。我们的方法的实证检验证明了它的适用性，上述情况下，并证实了我们的理论研究结果。
The Dynamical Anatomy of Anderson Acceleration:From Adaptive Momentum to Variable-Mass ODEs
College of Mathematics and Statistics, Nanjing University of Information Science and Technology（南京信息工程大学数学与统计学院）；Delft Institute of Applied Mathematics, Delft University of Technology（代尔夫特理工大学应用数学研究所）
