On low complexity Acceleration Techniques for Randomized Optimization: Supplementary Online Material
Institute of Theoretical Computer Science, ETH Zürich（苏黎世联邦理工学院理论计算机科学研究所）
Sebastian U. Stich
备注：
15 pages, 9 figures; the main part without the appendix is a preprint of a conference publication that will appear at PPSN XIII (2014); the appendix reports the complete numerical data that could not fit into the main part
Recently it was shown by Nesterov (2011) that techniques form convex optimization can be used to successfully accelerate simple derivative-free randomized optimization methods. The appeal of those schemes lies in their low complexity, which is only $Θ(n)$ per iteration---compared to $Θ(n^2)$ for algorithms storing second-order information or covariance matrices. From a high-level point of view, those accelerated schemes employ correlations between successive iterates---a concept looking similar to the evolution path used in Covariance Matrix Adaptation Evolution Strategies (CMA-ES).
In this contribution, we (i) implement and empirically test a simple accelerated random search scheme (SARP). Our study is the first to provide numerical evidence that SARP can effectively be implemented with adaptive step size control and does not require access to gradient or advanced line search oracles. We (ii) try to empirically verify the supposed analogy between the evolution path and SARP. We propose an algorithm CMA-EP that uses only the evolution path to bias the search. This algorithm can be generalized to a family of low memory schemes, with complexity $Θ(mn)$ per iteration, following a recent approach by Loshchilov (2014). The study shows that the performance of CMA-EP heavily depends on the spectra of the objective function and thus it cannot accelerate as consistently as SARP.
On the fast convergence of minibatch heavy ball momentum
Operations Research and Industrial Engineering（运筹学与工业工程）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Mathematics（数学）；New York University（纽约大学）；Computer Science and Engineering（计算机科学与工程）
