使用计算存储器的深度神经网络的混合精度训练
Mixed-precision training of deep neural networks using computational memory
摘要:
深度神经网络通过在解决图像和语音识别等许多现实问题时提供前所未有的类人性能,彻底改变了机器学习领域。然而,大型DNN的训练是一项计算密集型任务,这需要开发针对该应用的新型计算架构。其中电阻性存储器设备被组织在交叉杆阵列中的计算存储器单元可以用于在其电导状态中本地存储突触权重。昂贵的乘法累加操作可以使用基尔霍夫电路定律以非冯·诺依曼的方式在适当的位置执行。然而,一个关键挑战仍然是无法在权重更新过程中以可靠的方式改变设备的电导状态。我们提出了一种混合精度架构,它结合了一个计算存储器单元存储的突触权重与数字处理单元和一个额外的存储器单元积累权重更新的高精度。新架构提供的分类精度与浮点实现的精度相当,而不受新兴电阻存储器的非理想权重更新特性相关挑战的约束。一个两层的神经网络,其中的计算存储单元实现使用非线性随机模型的相变存储器设备达到测试准确率为97.40%的MNIST手写数字分类问题。
英文摘要:
Deep neural networks have revolutionized the field of machine learning by providing unprecedented human-like performance in solving many real-world problems such as image and speech recognition. Training of large DNNs, however, is a computationally intensive task, and this necessitates the development of novel computing architectures targeting this application. A computational memory unit where resistive memory devices are organized in crossbar arrays can be used to locally store the synaptic weights in their conductance states. The expensive multiply accumulate operations can be performed in place using Kirchhoff's circuit laws in a non-von Neumann manner. However, a key challenge remains the inability to alter the conductance states of the devices in a reliable manner during the weight update process. We propose a mixed-precision architecture that combines a computational memory unit storing the synaptic weights with a digital processing unit and an additional memory unit accumulating weight updates in high precision. The new architecture delivers classification accuracies comparable to those of floating-point implementations without being constrained by challenges associated with the non-ideal weight update characteristics of emerging resistive memories. A two layer neural network in which the computational memory unit is realized using non-linear stochastic models of phase-change memory devices achieves a test accuracy of 97.40% on the MNIST handwritten digit classification problem.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。