As the vocabulary size of modern word-based language models becomes ever larger, many sampling-based training criteria are proposed and investigated. The essence of these sampling methods is that the softmax-related traversal over the entire vocabulary can be simplified, giving speedups compared to the baseline. A problem we notice about the current landscape of such sampling methods is the lack of a systematic comparison and some myths about preferring one over another. In this work, we consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation. Linking back to the three traditional criteria, namely mean squared error, binary cross-entropy, and cross-entropy, we derive the theoretical solutions to the training problems. Contrary to some common belief, we show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities. Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim, with all sampling-based methods showing similar perplexities and word error rates while giving the expected speedups.
Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling
University of California, Los Angeles(加州大学洛杉矶分校);Amazon(亚马逊);Zhejiang University(浙江大学)
本文提出了一种用于学习图像的基于能量的生成式ConvNet模型的多重网格方法。对于每个网格,我们学习一个基于能量的概率模型,其中能量函数由自底向上的卷积神经网络(ConvNet或CNN)定义。学习这样的模型需要从模型中生成合成的示例。在我们的学习算法的每次迭代中,对于每个观察到的训练图像,我们通过从训练图像的最小1 x 1版本初始化有限步MCMC采样来在多个网格处生成合成图像。在每个后续网格的合成图像是由有限步MCMC初始化从合成图像生成的前一个粗网格。在得到合成实例后,根据合成实例与观测实例之间的差异,分别和同时更新多个网格处的模型参数。我们证明了这种多重网格方法可以学习现实的基于能量的生成ConvNet模型,并且它优于原始的对比发散(CD)和持久CD。
Effect of choice of probability distribution, randomness, and search methods for alignment modeling in sequence-to-sequence text-to-speech synthesis using hard alignment
1 National Institute of Informatics, Japan 2 SOKENDAI, Japan