一种快速简单的神经概率语言模型训练算法
A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
摘要：
尽管神经概率语言模型（NPLM）具有优越的性能，但由于其众所周知的长训练时间，即使对于中等大小的数据集也需要数周的时间，因此其使用远不如n-gram模型广泛。训练NPLM在计算上是昂贵的，因为它们是显式归一化的，这导致在计算对数似然梯度时必须考虑词汇表中的所有单词。我们提出了一个快速，简单的算法训练NPLM的基础上噪声对比估计，一个新引入的程序估计非归一化连续分布。我们研究了该算法在Penn Treebank语料库上的行为，并表明它将训练时间减少了一个数量级以上，而不会影响所得模型的质量。该算法也比重要性采样更有效和更稳定，因为它需要少得多的噪声样本来良好地执行。我们证明了所提出的方法的可扩展性，通过训练几个神经语言模型上的47万字语料库与80 K字的词汇，获得最先进的结果，微软研究句子完成挑战数据集。
英文摘要：
In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
