arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.00762stat.MLcs.LG
收藏

基于学习字典的多尺度序列建模

Multiscale sequence modeling with a learned dictionary

MILA(蒙特利尔学习算法研究所);IIT Kanpur(印度理工学院坎普尔分校);Google Brain(谷歌大脑)

Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, Yoshua Bengio

摘要:

我们提出了一个推广的神经网络序列模型。我们的多尺度模型不是一次预测一个符号,而是对多个可能重叠的多符号令牌进行预测。使用字节对编码(BPE)压缩算法的变体来学习用于训练模型的令牌字典。当应用于语言建模时,我们的模型具有字符级模型的灵活性,同时保持了单词级模型的许多性能优势。我们的实验表明,该模型在语言建模任务上的表现优于常规LSTM,特别是对于较小的模型。

英文摘要:

We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that the model is trained with. When applied to language modelling, our model has the flexibility of character-level models while maintaining many of the performance benefits of word-level models. Our experiments show that this model performs better than a regular LSTM on language modeling tasks, especially for smaller models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑