arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.09997math.OCcs.LGstat.ML
收藏

机器学习的拟牛顿法:忘记过去,只是样本

Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample

University of Michigan(密歇根大学);Lehigh University(利哈伊大学);KAUST(阿卜杜拉国王科技大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)

Albert S. Berahas, Majid Jahani, Peter Richtárik, Martin Takáč

摘要:

我们提出了两个采样拟牛顿法(采样LBFGS和采样LSR 1)解决机器学习中出现的经验风险最小化问题。与这些方法的经典变体相反,随着优化的进行,这些方法顺序地构建Hessian或逆Hessian近似,我们提出的方法在每次迭代时随机地在当前样本点周围样本点以产生这些近似。因此,构造的近似值利用更可靠的(最近的和本地的)信息,并且不依赖于可能显著陈旧的过去的信息。我们提出的算法是有效的访问数据点(时代),并有足够的并发性,以利用并行/分布式计算环境。我们为我们提出的方法提供了收敛保证。对玩具分类问题以及流行的基准二进制分类和神经网络训练任务的数值测试表明,该方法优于其经典变体。

英文摘要:

We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning. Contrary to the classical variants of these methods that sequentially build Hessian or inverse Hessian approximations as the optimization progresses, our proposed methods sample points randomly around the current iterate at every iteration to produce these approximations. As a result, the approximations constructed make use of more reliable (recent and local) information, and do not depend on past iterate information that could be significantly stale. Our proposed algorithms are efficient in terms of accessed data points (epochs) and have enough concurrency to take advantage of parallel/distributed computing environments. We provide convergence guarantees for our proposed methods. Numerical tests on a toy classification problem as well as on popular benchmarking binary classification and neural network training tasks reveal that the methods outperform their classical variants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑