arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.06170cs.CLcs.LG
收藏

Transformer on a Diet(减肥中的变形金刚)

Transformer on a Diet

Amazon Web Services(亚马逊云服务)

Chenguang Wang, Zihao Ye, Aston Zhang, Zheng Zhang, Alexander J. Smola

摘要:

由于Transformer能够以有效的方式捕获序列信息,因此它得到了广泛的应用。然而,最近的发展,如BERT和GPT-2,只提供了重型架构,重点是有效性。在本文中,我们探讨了三种精心设计的轻型Transformer架构,以确定计算量较少的Transformer是否可以产生有竞争力的结果。在语言模型基准数据集上的实验结果表明,这种折衷是有希望的,轻量级的Transformer最多减少了70%的参数,同时获得了与标准Transformer相比具有竞争力的困惑度。源代码是公开的。

英文摘要:

Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑