神经语言建模的自适应输入表示
Adaptive Input Representations for Neural Language Modeling
摘要:
我们引入了用于神经语言建模的自适应输入表示,将Grave等人(2017)的自适应softmax扩展到可变容量的输入表示。关于如何分解输入和输出层,以及是否建模单词、字符或子单词单元,有几种选择。我们进行了一个系统的比较流行的选择自我注意的架构。我们的实验表明,配备自适应嵌入的模型的训练速度是流行的字符输入CNN的两倍多,同时具有较低的参数数量。在WikiText-103基准测试中,我们实现了18.7的困惑度,与之前发布的最佳结果相比,提高了10.5的困惑度,而在十亿字基准测试中,我们实现了23.02的困惑度。
英文摘要:
We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。