一种用于神经语言建模的随机流言BMUF过程
A Random Gossip BMUF Process for Neural Language Modeling
摘要：
神经网络语言模型（NNLM）是工业自动识别系统的重要组成部分。训练NNLM的一个重要挑战是在扩展学习过程和处理大数据之间进行平衡。诸如块动量的常规方法提供了逐块模型更新滤波（BMUF）过程，并且在语音识别的性能没有降低的情况下实现了几乎线性的加速比。但是，它需要计算来自所有计算节点的模型平均值（例如，GPU），并且当计算节点的数量很大时，学习遭受严重的通信延迟。因此，BMUF在受限网络条件下不适用。在本文中，我们提出了一个分散的BMUF过程，其中模型被分成不同的组件，每个组件通过与一些随机选择的具有相同组件的邻居节点进行通信来更新，然后进行类似BMUF的过程。我们将这种方法应用于几个LSTM语言建模任务。实验结果表明，我们的方法实现了一贯优于传统的BMUF的性能。特别是，我们获得了较低的困惑比单GPU基线上的维基文本103基准使用4个GPU。此外，当扩展到8和16个GPU时，没有观察到性能下降。
英文摘要：
Neural network language model (NNLM) is an essential component of industrial ASR systems. One important challenge of training an NNLM is to leverage between scaling the learning process and handling big data. Conventional approaches such as block momentum provides a blockwise model update filtering (BMUF) process and achieves almost linear speedups with no performance degradation for speech recognition. However, it needs to calculate the model average from all computing nodes (e.g., GPUs) and when the number of computing nodes is large, the learning suffers from the severe communication latency. As a consequence, BMUF is not suitable under restricted network conditions. In this paper, we present a decentralized BMUF process, in which the model is split into different components, each of which is updated by communicating to some randomly chosen neighbor nodes with the same component, followed by a BMUF-like process. We apply this method to several LSTM language modeling tasks. Experimental results show that our approach achieves consistently better performance than conventional BMUF. In particular, we obtain a lower perplexity than the single-GPU baseline on the wiki-text-103 benchmark using 4 GPUs. In addition, no performance degradation is observed when scaling to 8 and 16 GPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
