arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1910.10073cs.CLcs.LG
收藏

深度自适应Transformer

Depth-Adaptive Transformer

Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Facebook AI Research(Facebook人工智能研究)

Maha Elbayad, Jiatao Gu, Edouard Grave, Michael Auli

摘要:

用于大规模任务的现有技术的序列到序列模型对每个输入序列执行固定数量的计算,而不管它是容易还是难以处理。在本文中,我们训练Transformer模型,它可以在网络的不同阶段进行输出预测,我们研究了不同的方法来预测一个特定的序列需要多少计算。与Universal Transformers中迭代应用同一组层的动态计算不同,我们在每一步应用不同的层来调整计算量和模型容量。在IWITOS德语-英语翻译上,我们的方法与调好的基线Transformer的准确性相匹配,同时使用不到四分之一的解码器层。

英文摘要:

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well tuned baseline Transformer while using less than a quarter of the decoder layers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑