arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.01736cs.LG
收藏

超越人类水平的准确性：深度学习中的计算挑战

Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

Baidu Research（百度研究院）

Joel Hestness, Newsha Ardalani, Greg Diamos

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

摘要：

深度学习（DL）研究从模型架构的变化和规模（更大的数据集和模型以及更多的计算）中获得了准确性和产品改进。对于硬件设计，很难预测DL模型的变化。然而，最近的先前工作表明，随着数据集大小的增长，DL模型的准确性和模型大小可预测地增长。本文利用先前的工作来预测数据集和模型大小的增长，以将DL准确性提高到人类水平以上，达到机器学习专家定义的前沿目标。数据集将需要增长$33$ -- $971 \times$，而模型将需要增长$6.6$ -- $456\times$以达到目标精度。我们进一步表征和项目的计算要求，训练这些应用程序的规模。我们的表征揭示了递归神经网络（RNN）的DL训练挑战的重要细分，与深度卷积网络的先前研究形成对比。RNN将具有相对适中的操作强度和非常大的内存占用需求。与新兴的加速器设计相比，大规模RNN训练特征表明设计具有更大的内存容量和片上缓存。

英文摘要：

Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware design, it is difficult to predict DL model changes. However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably. This paper leverages the prior work to project the dataset and model size growth required to advance DL accuracy beyond human-level, to frontier targets defined by machine learning experts. Datasets will need to grow $33$--$971 \times$, while models will need to grow $6.6$--$456\times$ to achieve target accuracies. We further characterize and project the computational requirements to train these applications at scale. Our characterization reveals an important segmentation of DL training challenges for recurrent neural networks (RNNs) that contrasts with prior studies of deep convolutional networks. RNNs will have comparatively moderate operational intensities and very large memory footprint requirements. In contrast to emerging accelerator designs, large-scale RNN training characteristics suggest designs with significantly larger memory capacity and on-chip caches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑