arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.02860cs.LGcs.CLstat.ML
收藏

Transformer -XL:固定长度上下文之外的注意语言模型

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Carnegie Mellon University(卡内基梅隆大学);Google Brain(谷歌大脑)

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov

摘要:

Transformers具有学习长期依赖的潜力,但在语言建模的设置中受到固定长度上下文的限制。我们提出了一种新的神经架构transformer-XL,使学习的依赖性超过一个固定的长度,而不会破坏时间的连贯性。它包括一个段级递归机制和一个新的位置编码方案。我们的方法不仅能够捕获长期依赖,而且还解决了上下文碎片问题。因此,Transformer-XL学习的依赖性比RNN长80%,比vanilla Transformers长450%,在短序列和长序列上都实现了更好的性能,并且在评估过程中比vanilla Transformers快1,800多倍。值得注意的是,我们将bpc/困惑度的最新结果提高到enwiki 8上的0.99,text 8上的1.08,WikiText-103上的18.3,One Billion Word上的21.8,以及Penn Treebank上的54.5(没有微调)。当只在WikiText-103上训练时,Transformer-XL设法生成具有数千个令牌的合理连贯,新颖的文本文章。我们的代码、预训练模型和超参数在Tensorflow和PyTorch中都可用。

英文摘要:

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑