In our previous work we have shown that resistive cross point devices, so called Resistive Processing Unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. We note that the models trained here are roughly 1500 times larger than the fully connected network trained on MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch. Thus, here we attempt to study the validity of the RPU approach for large scale networks.
Multi-level Binarized LSTM in EEG Classification for Wearable Devices
School of Electrical and Computer Engineering, University of Tehran（德黑兰大学电气与计算机工程学院）；Division of Intelligent Future Technologies, Malardalen University（马拉达伦大学智能未来技术系）
Unsupervised local learning based on voltage-dependent synaptic plasticity for resistive and ferroelectric synapses
Institut Interdisciplinaire d′Innovation Technologique (3IT)（跨学科技术创新研究所）；Université de Sherbrooke（谢布罗克大学）；Laboratoire Nanotechnologies Nanosystèmes (LN2)-IRL3463（纳米技术与纳米系统实验室）；CNRS（法国国家科学研究中心）；INSA Lyon（里昂国立应用科学学院）；Ecole Centrale de Lyon（里昂中央理工大学）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；STMicroelectronics（STMicroelectronics公司）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CEA（法国原子能委员会）；Grenoble INP（格勒诺布尔国立应用科学学院）；SPINTEC（SPINTEC研究中心）；IBM Research GmbH - Zurich Research Laboratory（IBM瑞士研究实验室）；IBM Research GmbH - Zurich Res（IBM瑞士研究实验室）
Efficient Split Learning LSTM Models for FPGA-based Edge IoT Devices
The Abdus Salam International Centre for Theoretical Physics（阿卜杜勒·萨拉姆国际理论物理中心）；Faculty of Technical Sciences, University of Novi Sad（诺威萨德大学技术科学学院）；The Institute for Artificial Intelligence Research and Development of Serbia（塞尔维亚人工智能研究与发展研究所）
