Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.
USDs: A universal stabilizer decoder framework using symmetry
Computer Science Program, Graduate School of Science and Technology, Meiji University（Meiji大学计算机科学系）；Department of Computer Science, School of Science and Technology, Meiji University（Meiji大学计算机科学系）
Stabilization Analysis and Mode Recognition of Kerosene Supersonic Combustion: A Deep Learning Approach Based on Res-CNN-beta-VAE
City University of Hong Kong（香港城市大学）；State Key Laboratory of High Temperature Gas Dynamics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所高温气体动力学国家重点实验室）
Trade-off between Gradient Measurement Efficiency and Expressivity in Deep Quantum Neural Networks
Quantum Laboratory, Fujitsu Research, Fujitsu Limited（Fujitsu 量子实验室、Fujitsu 研究、Fujitsu 有限公司）；Department of Applied Physics, Graduate School of Engineering, The University of Tokyo（东京大学工学系应用物理系）
