arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.09048cs.LGcs.AI
收藏

自私稀疏RNN训练

Selfish Sparse RNN Training

Eindhoven University of Technology(埃因霍温理工大学);University of Twente(特文特大学)

Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy

摘要:

稀疏神经网络已被广泛应用于减少训练和部署过参数化深度神经网络的计算需求。对于推理加速,从预训练的密集网络(密集到稀疏训练)中发现稀疏网络的方法有效。最近,已经提出了动态稀疏训练(DST)来训练稀疏神经网络,而不预先训练密集模型(稀疏到稀疏训练),使得训练过程也可以加速。然而,以前的稀疏到稀疏方法主要集中在多层感知器网络(MLP)和卷积神经网络(CNN)上,无法匹配递归神经网络(RNN)设置中的密集到稀疏方法的性能。在本文中,我们提出了一种方法,训练内在稀疏的RNN与一个固定的参数计数在一个单一的运行,而不影响性能。在训练过程中,我们允许RNN层在单元门之间进行非均匀的重新分布,以实现更好的正则化。此外,我们提出了SNT-ASGD,这是平均随机梯度优化器的一种新变体,它显着提高了RNN的所有稀疏训练方法的性能。使用这些策略,我们在Penn TreeBank和Wikitext-2数据集上使用各种类型的RNN实现了最先进的稀疏训练结果,优于密集到稀疏的方法。我们的代码可在https://github.com/Shiweiliuiiiiiii/Selfish-RNN上获得。

英文摘要:

Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks. For inference acceleration, methods that discover a sparse network from a pre-trained dense network (dense-to-sparse training) work effectively. Recently, dynamic sparse training (DST) has been proposed to train sparse neural networks without pre-training a dense model (sparse-to-sparse training), so that the training process can also be accelerated. However, previous sparse-to-sparse methods mainly focus on Multilayer Perceptron Networks (MLPs) and Convolutional Neural Networks (CNNs), failing to match the performance of dense-to-sparse methods in the Recurrent Neural Networks (RNNs) setting. In this paper, we propose an approach to train intrinsically sparse RNNs with a fixed parameter count in one single run, without compromising performance. During training, we allow RNN layers to have a non-uniform redistribution across cell gates for better regularization. Further, we propose SNT-ASGD, a novel variant of the averaged stochastic gradient optimizer, which significantly improves the performance of all sparse training methods for RNNs. Using these strategies, we achieve state-of-the-art sparse training results, better than the dense-to-sparse methods, with various types of RNNs on Penn TreeBank and Wikitext-2 datasets. Our codes are available at https://github.com/Shiweiliuiiiiiii/Selfish-RNN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑