arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1603.07341cs.LGcs.NEstat.ML
收藏

利用电阻交叉点器件加速深度神经网络训练

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices

IBM T.J. Watson Research Center(IBM托马斯·J·沃森研究中心)

Tayfun Gokmen, Yurii Vlasov

摘要:

近年来,深度神经网络(DNN)在大规模分析和分类任务(如语音识别、视觉目标检测、模式提取等)中表现出了显著的业务影响。然而,大型DNN的训练普遍被认为是耗时且计算密集型的任务,需要企业级计算资源招募许多天。在这里,我们提出了一个电阻处理单元(RPU)设备的概念,它可以潜在地将DNN训练加速几个数量级,同时使用更少的功率。所提出的RPU设备可以存储和更新本地的权重值,从而最大限度地减少训练过程中的数据移动,并允许充分利用本地和并行的训练算法。我们确定了RPU设备和系统规范,用于在现实的CMOS兼容技术中实现DNN训练的加速器芯片。对于具有约10亿个权重的大型DNN,与最先进的微处理器相比,这种大规模并行RPU架构可以实现30,000倍的加速因子,同时提供84,000 GigaOps/s/W的功率效率。目前需要在拥有数千台机器的大型集群上进行数天培训的问题可以在单个RPU加速器上在数小时内解决。由RPU加速器集群组成的系统将能够解决具有数万亿参数的大数据问题,这些参数是当今无法解决的,例如,自然语音识别和所有世界语言之间的翻译,对大量业务和科学数据流的实时分析,来自大量物联网传感器的多模式感知数据流的集成和分析。

英文摘要:

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We identify the RPU device and system specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30,000X compared to state-of-the-art microprocessors while providing power efficiency of 84,000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisted of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration and analysis of multimodal sensory data flows from massive number of IoT (Internet of Things) sensors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑