二进制网络中通过流水线截断误差反向传播的硬件高效在线学习
Hardware-efficient on-line learning through pipelined truncated-error backpropagation in binary-state networks
摘要：
使用反向传播训练的人工神经网络（ANN）是功能强大的学习架构，在各种基准测试中具有最先进的性能。大量的努力致力于开发定制的硅器件，以加速人工神经网络的推理。然而，加速培训阶段却没有引起多少关注。在本文中，我们描述了一种基于流水线反向传播的前馈多层人工神经网络的硬件高效在线学习技术。学习与前向传递中的推理并行执行，消除了对显式后向传递的需要，并且不需要额外的权重查找。通过在前馈网络中使用二进制状态变量和在截断误差反向传播中使用三进制误差，消除了在向前和向后传递中对任何乘法的需要，并且大大降低了流水线的存储器要求。由于前向神经和反向传播误差信号路径中的稀疏性，加法运算的进一步减少有助于高效的硬件实现。对于概念验证，我们演示了在线学习的MNIST手写数字分类斯巴达6 FPGA接口与外部1Gb DDR2 DRAM，这表明小退化的测试误差性能相比，同等大小的二进制人工神经网络训练离线使用标准的反向传播和准确的错误。我们的研究结果强调了流水线反向传播和二进制状态网络之间的协同作用，大大减少了计算和内存需求，使流水线在线学习在深度网络中变得实用。
英文摘要：
Artificial neural networks (ANNs) trained using backpropagation are powerful learning architectures that have achieved state-of-the-art performance in various benchmarks. Significant effort has been devoted to developing custom silicon devices to accelerate inference in ANNs. Accelerating the training phase, however, has attracted relatively little attention. In this paper, we describe a hardware-efficient on-line learning technique for feedforward multi-layer ANNs that is based on pipelined backpropagation. Learning is performed in parallel with inference in the forward pass, removing the need for an explicit backward pass and requiring no extra weight lookup. By using binary state variables in the feedforward network and ternary errors in truncated-error backpropagation, the need for any multiplications in the forward and backward passes is removed, and memory requirements for the pipelining are drastically reduced. Further reduction in addition operations owing to the sparsity in the forward neural and backpropagating error signal paths contributes to highly efficient hardware implementation. For proof-of-concept validation, we demonstrate on-line learning of MNIST handwritten digit classification on a Spartan 6 FPGA interfacing with an external 1Gb DDR2 DRAM, that shows small degradation in test error performance compared to an equivalently sized binary ANN trained off-line using standard back-propagation and exact errors. Our results highlight an attractive synergy between pipelined backpropagation and binary-state networks in substantially reducing computation and memory requirements, making pipelined on-line learning practical in deep networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
