TensorDash is a hardware level technique for enabling data-parallel MAC units to take advantage of sparsity in their input operand streams. When used to compose a hardware accelerator for deep learning, TensorDash can speedup the training process while also increasing energy efficiency. TensorDash combines a low-cost, sparse input operand interconnect comprising an 8-input multiplexer per multiplier input, with an area-efficient hardware scheduler. While the interconnect allows a very limited set of movements per operand, the scheduler can effectively extract sparsity when it is present in the activations, weights or gradients of neural networks. Over a wide set of models covering various applications, TensorDash accelerates the training process by $1.95{\times}$ while being $1.89\times$ more energy-efficient, $1.6\times$ more energy efficient when taking on-chip and off-chip memory accesses into account. While TensorDash works with any datatype, we demonstrate it with both single-precision floating-point units and bfloat16.
DNN Training Acceleration via Exploring GPGPU Friendly Sparsity
Shanghai Jiao Tong University（上海交通大学）
深度神经网络~（DNN）的训练阶段消耗大量的处理时间和能量。利用DNN的稀疏性的压缩技术可以有效地加速DNN的推理阶段。然而，它几乎不用于训练阶段，因为训练阶段涉及使用图形处理器上的通用计算（GPGPU）的密集矩阵乘法，这支持规则和结构化的数据布局。本文首先提出了一种近似随机丢弃（Approximate Random Dropout）算法，该算法用规则的在线生成的基于行或基于块的丢弃模式代替传统的神经元和突触的随机丢弃，以消除多层感知器MLP和长短期记忆LSTM中不必要的计算和数据访问。然后，我们开发了一种基于SGD的搜索算法，该算法产生基于行或基于瓦片的丢弃模式的分布，以补偿潜在的准确性损失。此外，针对卷积神经网络CNN的训练加速问题，首先探讨了输入特征图的重要性和敏感性，然后提出了敏感性感知的丢弃方法，根据输入特征图的敏感性动态丢弃输入特征图，从而在保持较好的神经网络精度的同时，获得更大的前向和后向训练加速。为了便于DNN编程，我们构建了一个DNN训练计算框架，该框架将软件堆栈中的拟议技术统一起来。因此，GPGPU只需要支持基本运算符-矩阵乘法，并且无论DNN模型如何，都可以实现显着的性能提升。
SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training
School of Microelectronics, BDBC, Beihang University（北京航空航天大学微电子学院，大数据科学与脑机智能高精尖创新中心）；School of Computer Science and Engineering, BDBC, Beihang University（北京航空航天大学计算机科学与工程学院，大数据科学与脑机智能高精尖创新中心）；Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）
