arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.15045cs.CL
收藏

储槽式Transformers

Reservoir Transformers

UC Berkeley(加州大学伯克利分校);Facebook AI Research(脸书人工智能研究院)

Sheng Shen, Alexei Baevski, Ari S. Morcos, Kurt Keutzer, Michael Auli, Douwe Kiela

摘要:

我们证明,Transformers获得令人印象深刻的性能,即使当一些层被随机初始化,从来没有更新。受机器学习中旧的和成熟的想法的启发,我们探索了各种非线性“水库”层,穿插着常规的Transformer层,并在各种机器翻译和(掩码)语言建模任务上显示了在收敛之前的挂钟计算时间以及整体性能的改进。

英文摘要:

We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑