arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.10507cs.CLcs.SDeess.ASstat.ML
收藏

基于采样的神经语言模型训练准则研究

On Sampling-Based Training Criteria for Neural Language Modeling

RWTH Aachen University(亚琛工业大学);AppTek GmbH(AppTek公司)

Yingbo Gao, David Thulke, Alexander Gerstenberger, Khoa Viet Tran, Ralf Schlüter, Hermann Ney

摘要:

随着现代基于单词的语言模型的词汇量越来越大,许多基于采样的训练准则被提出和研究。这些采样方法的本质是,可以简化整个词汇表上与softmax相关的遍历,从而与基线相比提高了速度。我们注意到的一个问题是,这种抽样方法目前的情况是缺乏系统的比较和一些关于偏爱一个而不是另一个的神话。在这项工作中,我们考虑蒙特卡罗抽样,重要性采样,一种新的方法,我们称之为补偿部分求和,和噪声对比估计。链接回三个传统的标准,即均方误差,二进制交叉熵,交叉熵,我们得到的理论解决方案的训练问题。与一些常见的信念相反,我们表明,所有这些抽样方法可以同样好地执行,只要我们纠正预期的类后验概率。在Switchboard和LibriSpeech上的语言建模和自动语音识别的实验结果支持我们的主张,所有基于采样的方法都显示出类似的困惑和单词出错率,同时给出了预期的加速。

英文摘要:

As the vocabulary size of modern word-based language models becomes ever larger, many sampling-based training criteria are proposed and investigated. The essence of these sampling methods is that the softmax-related traversal over the entire vocabulary can be simplified, giving speedups compared to the baseline. A problem we notice about the current landscape of such sampling methods is the lack of a systematic comparison and some myths about preferring one over another. In this work, we consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation. Linking back to the three traditional criteria, namely mean squared error, binary cross-entropy, and cross-entropy, we derive the theoretical solutions to the training problems. Contrary to some common belief, we show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities. Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim, with all sampling-based methods showing similar perplexities and word error rates while giving the expected speedups.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑