This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations. The first occurs in distributed computing environments where some of the computational nodes devoted to the evaluation of the function and gradient are unable to return results on time. A similar challenge occurs in a multi-batch approach in which the data points used to compute function and gradients are purposely changed at each iteration to accelerate the learning process. Difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the updating process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, studies the convergence properties for both convex and nonconvex functions, and illustrates the behavior of the algorithm in a distributed computing platform on binary classification logistic regression and neural network training problems that arise in machine learning.
A family of spectral conjugate gradient algorithms derived by least-squares approximations based on a modified quasi--Newton update with application to a revised robust binary classification model
Free University of Bozen–Bolzano（博尔扎诺自由大学）；Semnan University（塞姆南大学）；American University（美利坚大学）
An Inexact Projected Gradient Method with Rounding and Lifting by Nonlinear Programming for Solving Rank-One Semidefinite Relaxation of Polynomial Optimization
Massachusetts Institute of Technology（麻省理工学院）；National University of Singapore（新加坡国立大学）
AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute for Information Transmission Problems RAS（俄罗斯科学院信息传输问题研究所）；Innopolis University（因诺波利斯大学）；Skoltech（斯科尔科沃科学技术研究院）
