arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.11417cs.LGcs.AI
收藏

Legendre记忆单元训练的优化

Parallelizing Legendre Memory Unit Training

Center for Theoretical Neuroscience, University of Waterloo(滑铁卢大学理论神经科学中心);Applied Brain Research(应用大脑研究(机构))

Narsimha Chilkuri, Chris Eliasmith

摘要:

最近,提出了一种名为Legendre Memory Unit(LMU)的新循环神经网络(RNN),并证明其在多个基准数据集上实现了最先进的性能。在这里,我们利用LMU的线性时不变(LTI)内存组件来构建一个简化的变体,该变体可以在训练期间并行化(但在推理期间作为RNN执行),从而克服了在GPU上训练RNN的众所周知的限制。我们证明了这种有助于并行化的重新表述,它可以普遍应用于任何循环分量为线性的深度网络,使训练速度提高200倍。其次,为了验证它的实用性,我们将其与原始LMU和各种已发布的LSTM和Transformer网络在七个基准上进行了性能比较,从psMNIST到情感分析再到机器翻译。我们证明了我们的模型在所有数据集上都表现出卓越的性能,通常使用更少的参数。例如,我们的LMU在psMNIST上设置了一个新的最先进的结果,并使用了一半的参数,同时在IMDB情感分析上优于DistilBERT和LSTM模型。

英文摘要:

Recently, a new recurrent neural network (RNN) named the Legendre Memory Unit (LMU) was proposed and shown to achieve state-of-the-art performance on several benchmark datasets. Here we leverage the linear time-invariant (LTI) memory component of the LMU to construct a simplified variant that can be parallelized during training (and yet executed as an RNN during inference), thus overcoming a well known limitation of training RNNs on GPUs. We show that this reformulation that aids parallelizing, which can be applied generally to any deep network whose recurrent components are linear, makes training up to 200 times faster. Second, to validate its utility, we compare its performance against the original LMU and a variety of published LSTM and transformer networks on seven benchmarks, ranging from psMNIST to sentiment analysis to machine translation. We demonstrate that our models exhibit superior performance on all datasets, often using fewer parameters. For instance, our LMU sets a new state-of-the-art result on psMNIST, and uses half the parameters while outperforming DistilBERT and LSTM models on IMDB sentiment analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑