通过解码过去改进语言建模
Improved Language Modeling by Decoding the Past
摘要：
高度正则化的LSTM在语言建模的几个基准数据集上取得了令人印象深刻的结果。我们提出了一种新的正则化方法，该方法基于使用下一个令牌的预测分布来解码上下文中的最后一个令牌。这使模型偏向于保留更多的上下文信息，从而提高了预测下一个令牌的能力。由于参数数量和训练时间的开销可以忽略不计，我们的Past Decode Regularization（PDR）方法使用单个softmax在Penn Treebank上实现了55.6的单词级困惑度，在WikiText-2数据集上实现了63.5的单词级困惑度。我们还显示了使用PDR与混合softmax相结合的收益，在这些数据集上实现了53.8和60.5的单词级困惑。此外，我们的方法实现了1.169位每字符的Penn Treebank字符数据集字符级语言建模。这些结果在各自的环境中构成了新的最先进水平。
英文摘要：
Highly regularized LSTMs achieve impressive results on several benchmark datasets in language modeling. We propose a new regularization method based on decoding the last token in the context using the predicted distribution of the next token. This biases the model towards retaining more contextual information, in turn improving its ability to predict the next token. With negligible overhead in the number of parameters and training time, our Past Decode Regularization (PDR) method achieves a word level perplexity of 55.6 on the Penn Treebank and 63.5 on the WikiText-2 datasets using a single softmax. We also show gains by using PDR in combination with a mixture-of-softmaxes, achieving a word level perplexity of 53.8 and 60.5 on these datasets. In addition, our method achieves 1.169 bits-per-character on the Penn Treebank Character dataset for character level language modeling. These results constitute a new state-of-the-art in their respective settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
