使用GPU的大规模随机学习
Large-Scale Stochastic Learning using GPUs
摘要：
在这项工作中，我们提出了一个加速随机学习系统非常大规模的应用。加速是通过将训练算法映射到大规模并行处理器上来实现的：我们展示了广泛使用的随机坐标下降/上升算法的并行异步GPU实现，该算法可以提供高达35倍的连续CPU实现速度。为了在不适合单个GPU内存的非常大的数据集上进行训练，我们考虑了分布式随机学习技术。当训练数据通过实例或特征分布时，我们提出了一种新的方法用于最优地聚集来自工作节点的模型更新。使用这种技术，我们证明了可以在多达8个工作节点上扩展随机学习，而不会损失任何显著的训练时间。最后，我们将GPU加速与优化的分布式方法相结合，在由2亿个训练样本和7500万个特征组成的数据集上进行训练。我们表明，通过跨4个GPU扩展，可以在大约4秒内实现高度的训练准确性：与跨4个CPU的多线程分布式实现相比，训练时间加快了20倍。
英文摘要：
In this work we propose an accelerated stochastic learning system for very large-scale applications. Acceleration is achieved by mapping the training algorithm onto massively parallel processors: we demonstrate a parallel, asynchronous GPU implementation of the widely used stochastic coordinate descent/ascent algorithm that can provide up to 35x speed-up over a sequential CPU implementation. In order to train on very large datasets that do not fit inside the memory of a single GPU, we then consider techniques for distributed stochastic learning. We propose a novel method for optimally aggregating model updates from worker nodes when the training data is distributed either by example or by feature. Using this technique, we demonstrate that one can scale out stochastic learning across up to 8 worker nodes without any significant loss of training time. Finally, we combine GPU acceleration with the optimized distributed method to train on a dataset consisting of 200 million training examples and 75 million features. We show by scaling out across 4 GPUs, one can attain a high degree of training accuracy in around 4 seconds: a 20x speed-up in training time compared to a multi-threaded, distributed implementation across 4 CPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
