arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.10447cs.LGcs.AIcs.CLcs.NE
收藏

线性时间内的Transformer质量

Transformer Quality in Linear Time

Cornell University(康奈尔大学);Google Research(谷歌研究院)

Weizhe Hua, Zihang Dai, Hanxiao Liu, Quoc V. Le

摘要:

我们重新审视了Transformers的设计选择,并提出了解决其在处理长序列时的弱点的方法。首先,我们提出了一个简单的层命名为门控注意力单元,它允许使用一个较弱的单头注意力与最小的质量损失。然后,我们提出了一个线性近似方法补充这个新的层,这是加速器友好的和极具竞争力的质量。所得到的模型名为FLASH,在短(512)和长(8 K)上下文长度上匹配改进的Transformers的困惑,在Wiki-40 B上实现高达4.9 $\times$的训练加速,在PG-19上实现12.1 $\times$的自回归语言建模,在C4上实现4.8 $\times$的掩蔽语言建模。

英文摘要:

We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9$\times$ on Wiki-40B and 12.1$\times$ on PG-19 for auto-regressive language modeling, and 4.8$\times$ on C4 for masked language modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑