arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.09113cs.LGcs.DCmath.OC
收藏

GPU加速子采样牛顿法

GPU Accelerated Sub-Sampled Newton's Method

Purdue Univ.(普渡大学);Univ. of Queensland(昆士兰大学);ICSI(国际计算机科学研究所);Univ. of California at Berkeley(加州大学伯克利分校)

Sudhir B. Kylasa, Farbod Roosta-Khorasani, Michael W. Mahoney, Ananth Grama

摘要:

仅依赖于梯度信息的一阶方法通常用于各种机器学习(ML)和数据分析(DA)应用中。这归因于其实现的简单性,以及每次迭代的计算/存储成本低。然而,他们遭受显着的缺点,最值得注意的是,他们的性能下降,越来越多的问题病态。此外,它们通常涉及大量的超参数,并且众所周知地对诸如步长之类的参数敏感。通过结合来自Hessian的额外信息,二阶方法已被证明对许多此类对抗性影响具有弹性。然而,使用曲率信息的这些优点是以更高的每次迭代成本为代价的,这在\enquote{big data}机制中可能在计算上是禁止的。在本文中,我们表明,与传统的信念相反,二阶方法,当实施得当,可以更有效地比一阶替代品在许多大规模ML/ DA应用。特别是,在凸设置中,我们考虑经典的牛顿\textsf{'}的方法,其中的海森和/或梯度随机子采样的变种。我们表明,通过有效地利用GPU的功能,这种随机牛顿型算法可以显着加速,并且可以轻松地超越流行ML/ DA软件包(如TensorFlow)中现有技术的最新实现。此外,与一阶方法相比,这些随机化方法产生较小的内存开销。特别是,我们表明,对于百万维的问题,我们的GPU加速子采样牛顿\textsf{'}的方法实现了更高的测试精度在毫秒相比,几十秒的一阶替代品。

英文摘要:

First order methods, which solely rely on gradient information, are commonly used in diverse machine learning (ML) and data analysis (DA) applications. This is attributed to the simplicity of their implementations, as well as low per-iteration computational/storage costs. However, they suffer from significant disadvantages; most notably, their performance degrades with increasing problem ill-conditioning. Furthermore, they often involve a large number of hyper-parameters, and are notoriously sensitive to parameters such as the step-size. By incorporating additional information from the Hessian, second-order methods, have been shown to be resilient to many such adversarial effects. However, these advantages of using curvature information come at the cost of higher per-iteration costs, which in \enquote{big data} regimes, can be computationally prohibitive. In this paper, we show that, contrary to conventional belief, second-order methods, when implemented appropriately, can be more efficient than first-order alternatives in many large-scale ML/ DA applications. In particular, in convex settings, we consider variants of classical Newton\textsf{'}s method in which the Hessian and/or the gradient are randomly sub-sampled. We show that by effectively leveraging the power of GPUs, such randomized Newton-type algorithms can be significantly accelerated, and can easily outperform state of the art implementations of existing techniques in popular ML/ DA software packages such as TensorFlow. Additionally these randomized methods incur a small memory overhead compared to first-order methods. In particular, we show that for million-dimensional problems, our GPU accelerated sub-sampled Newton\textsf{'}s method achieves a higher test accuracy in milliseconds as compared with tens of seconds for first order alternatives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑