arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1903.03129cs.DCcs.LG
收藏

SLIDE:为大规模深度学习系统的智能算法而非硬件加速辩护

SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Rice University(莱斯大学);Intel Corporation(英特尔公司)

Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

摘要:

深度学习(DL)算法是现代机器学习系统的中心焦点。随着数据量的不断增长,训练具有数亿参数的大型神经网络以保持足够的容量来存储这些数据量并获得最先进的精度已经成为惯例。为了避免与大型模型和数据相关的昂贵计算,社区越来越多地投资于用于模型训练的专用硬件。然而,专门的硬件是昂贵的,很难推广到众多的任务。算法方面的进展未能证明其相对于NVIDIA-V100 GPU等强大硬件的直接优势。本文提供了一个例外。我们提出了SLIDE(Sub-LInear深度学习引擎),它独特地将智能随机算法与多核并行和工作负载优化相结合。仅使用CPU,SLIDE就大大减少了训练和推理过程中的计算量,优于在最佳GPU上优化的Tensorflow(TF)实现。我们对具有大型全连接架构的行业规模推荐数据集进行的评估表明,在任何给定的精度水平下,在44核CPU上使用SLIDE进行训练的速度都比在Tesla V100上使用TF训练的相同网络快3.5倍以上(1小时对3.5小时)。在相同的CPU硬件上,SLIDE比TF快10倍以上。我们提供代码和脚本的可重复性。

英文摘要:

Deep Learning (DL) algorithms are the central focus of modern machine learning systems. As data volumes keep growing, it has become customary to train large neural networks with hundreds of millions of parameters to maintain enough capacity to memorize these volumes and obtain state-of-the-art accuracy. To get around the costly computations associated with large models and data, the community is increasingly investing in specialized hardware for model training. However, specialized hardware is expensive and hard to generalize to a multitude of tasks. The progress on the algorithmic front has failed to demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs. This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning Engine) that uniquely blends smart randomized algorithms, with multi-core parallelism and workload optimization. Using just a CPU, SLIDE drastically reduces the computations during both training and inference outperforming an optimized implementation of Tensorflow (TF) on the best available GPU. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF. We provide codes and scripts for reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑