arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1610.09893cs.CLcs.LG
收藏

LightRNN:高效存储和计算的循环神经网络

LightRNN: Memory and Computation-Efficient Recurrent Neural Networks

Nanjing University of Science and Technology(南京理工大学);Microsoft Research Asia(微软亚洲研究院)

Xiang Li, Tao Qin, Jian Yang, Tie-Yan Liu

摘要:

循环神经网络(RNN)在语言建模和机器翻译等自然语言处理任务中取得了最先进的性能。然而,当词汇量很大时,RNN模型将变得非常大(例如,可能超出GPU设备的存储器容量),并且其训练将变得非常低效。在这项工作中,我们提出了一种新的技术来应对这一挑战。关键思想是使用2-分量(2C)共享嵌入来表示单词。我们将词汇表中的每个单词分配到一个表中,其中的每一行与一个向量关联,每一列与另一个向量关联。根据单词在表中的位置,单词由两个分量共同表示:行向量和列向量。由于同一行中的单词共享行向量,而同一列中的单词共享列向量,因此我们仅需要$2 \sqrt{|V|}$向量来表示$|V|$唯一单词的词汇表,这远少于现有方法所需的$|V|$向量。基于2-Component共享嵌入,我们设计了一种新的RNN算法,并在几个基准数据集上使用语言建模任务对其进行了评估。结果表明,我们的算法显着减少了模型的大小,加快了训练过程,而不牺牲准确性(它实现了类似的,如果不是更好,困惑相比,最先进的语言模型)。值得注意的是,在10亿字基准数据集上,我们的算法实现了与以前的语言模型相当的复杂度,同时将模型大小减少了40-100倍,并将训练过程加快了2倍。我们命名我们提出的算法\emph{LightRNN},以反映其非常小的模型大小和非常高的训练速度。

英文摘要:

Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need $2 \sqrt{|V|}$ vectors to represent a vocabulary of $|V|$ unique words, which are far less than the $|V|$ vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm \emph{LightRNN} to reflect its very small model size and very high training speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑