RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments can be prohibitive in high dimensional problems. Application of second order methods, on the other hand, is impracticable because computation of objective function Hessian inverses incurs excessive computational cost. BFGS modifies gradient descent by introducing a Hessian approximation matrix computed from finite gradient differences. RES utilizes stochastic gradients in lieu of deterministic gradients for both, the determination of descent directions and the approximation of the objective function's curvature. Since stochastic gradients can be computed at manageable computational cost RES is realizable and retains the convergence rate advantages of its deterministic counterparts. Convergence results show that lower and upper bounds on the Hessian egeinvalues of the sample functions are sufficient to guarantee convergence to optimal arguments. Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS. An application of RES to the implementation of support vector machines is developed.
Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
University of British Columbia（英属哥伦比亚大学）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Element AI（元素人工智能公司）；CCAI Affiliate Chair (Amii)（加拿大先进人工智能研究所CCAI附属主席）
Statistically adaptive learning for a general class of cost functions (SA L-BFGS)
Cornell University（康奈尔大学）；Context Relevant, Inc.（Context Relevant公司）；University of Washington（华盛顿大学）
我们提出了一个系统，可以快速进行万亿级机器学习的模型实验，具有数万亿个非零特征，数十亿个训练样本和数百万个参数。我们对文献的贡献是一种新的方法（SA L-BFGS），通过使用统计工具来平衡以前的权重，旧的训练样本和新的训练样本的贡献，以实现快速收敛，迭代次数很少，从而改变批量L-BFGS，以近实时地执行。据我们所知，其结果是文献中报道的最具可扩展性和灵活性的线性学习系统，击败了目前最好的系统（Vowpal Wabbit和AllReduce）的标准实践。使用来自腾讯公司的KDD Cup 2012数据集。我们提供了实验结果来验证这种方法的性能。
Optimizing the optimizer for data driven deep neural networks and physics informed neural networks
CSIRO Data61（联邦科学与工业研究组织Data61）；College of Engineering and Computer Science, The Australian National University（澳大利亚国立大学工程与计算机科学学院）；Defence Science and Technology Group, Department of Defence（国防部国防科学与技术集团）
An Inexact Projected Gradient Method with Rounding and Lifting by Nonlinear Programming for Solving Rank-One Semidefinite Relaxation of Polynomial Optimization
Massachusetts Institute of Technology（麻省理工学院）；National University of Singapore（新加坡国立大学）
A family of spectral conjugate gradient algorithms derived by least-squares approximations based on a modified quasi--Newton update with application to a revised robust binary classification model
Free University of Bozen–Bolzano（博尔扎诺自由大学）；Semnan University（塞姆南大学）；American University（美利坚大学）
