arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.08720cs.LGcs.AIcs.CLstat.ML
收藏

递归序列模型中的预定义稀疏性

Predefined Sparseness in Recurrent Sequence Models

Ghent University(根特大学);imec(imec 微电子研究中心)

Thomas Demeester, Johannes Deleu, Fréderic Godin, Chris Develder

摘要:

在训练神经网络时引入稀疏性已被证明可以产生具有较低内存占用但与密集模型相似的有效性的模型。然而,稀疏性通常是从密集模型开始诱导的,因此这种优势在训练过程中并不存在。我们提出了在NLP应用程序的递归序列模型中预先实施稀疏性的技术,也有利于训练。首先,在语言建模中,我们展示了如何在不增加参数数量的情况下增加递归层中的隐藏状态大小,从而获得更具表达力的模型。其次,对于序列标签,我们表明,词嵌入与预定义的稀疏导致类似的性能密集嵌入,在一小部分的可训练参数的数量。

英文摘要:

Inducing sparseness while training neural networks has been shown to yield models with a lower memory footprint but similar effectiveness to dense models. However, sparseness is typically induced starting from a dense model, and thus this advantage does not hold during training. We propose techniques to enforce sparseness upfront in recurrent sequence models for NLP applications, to also benefit training. First, in language modeling, we show how to increase hidden state sizes in recurrent layers without increasing the number of parameters, leading to more expressive models. Second, for sequence labeling, we show that word embeddings with predefined sparseness lead to similar performance as dense embeddings, at a fraction of the number of trainable parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑