arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.00993cs.CLstat.ML
收藏

基于动态噪声对比度估计的神经元跨维随机场语言模型的改进训练

Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation

Tsinghua University(清华大学)

Bin Wang, Zhijian Ou

摘要:

本文提出了一种新的全句语言模型--神经跨维随机场语言模型(neural TRF LM),它将句子建模为随机场的集合,势函数由一个神经网络定义。在本文中,我们扩展了NCE,并提出了动态噪声对比估计(DNCE)来解决NCE训练中观察到的两个问题。首先,引入动态噪声分布并同时训练以收敛到数据分布。这有助于显著减少NCE中使用的噪声样本数量,并降低训练成本。第二,DNCE区分从噪声分布生成的句子和从数据分布和噪声分布的插值生成的句子。这消除了由训练集的稀疏性引起的过拟合问题。利用DNCE,我们可以在大规模语料库(约8亿字)和大词汇量(约568 K字)上成功有效地训练神经TRF LM。神经TRF LM与LSTM LM一样好,参数更少,并且在重新评分句子时快5倍~ 114倍。用LSTM LM和n-gram LM插值神经TRF LM可以进一步降低出错率。

英文摘要:

A new whole-sentence language model - neural trans-dimensional random field language model (neural TRF LM), where sentences are modeled as a collection of random fields, and the potential function is defined by a neural network, has been introduced and successfully trained by noise-contrastive estimation (NCE). In this paper, we extend NCE and propose dynamic noise-contrastive estimation (DNCE) to solve the two problems observed in NCE training. First, a dynamic noise distribution is introduced and trained simultaneously to converge to the data distribution. This helps to significantly cut down the noise sample number used in NCE and reduce the training cost. Second, DNCE discriminates between sentences generated from the noise distribution and sentences generated from the interpolation of the data distribution and the noise distribution. This alleviates the overfitting problem caused by the sparseness of the training set. With DNCE, we can successfully and efficiently train neural TRF LMs on large corpus (about 0.8 billion words) with large vocabulary (about 568 K words). Neural TRF LMs perform as good as LSTM LMs with less parameters and being 5x~114x faster in rescoring sentences. Interpolating neural TRF LMs with LSTM LMs and n-gram LMs can further reduce the error rates.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑