arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.11346cs.LG
收藏

次线性记忆:如何让表演者成为SLiM

Sub-Linear Memory: How to Make Performers SLiM

University of Cambridge(剑桥大学);Google Brain(谷歌大脑);Columbia University(哥伦比亚大学);DeepMind(深度思维);Stanford University(斯坦福大学);Alan Turing Institute(艾伦·图灵研究所)

Valerii Likhosherstov, Krzysztof Choromanski, Jared Davis, Xingyou Song, Adrian Weller

摘要:

Transformer架构彻底改变了对序列数据的深度学习,在各种应用的最先进解决方案中无处不在。然而普通的Transformers是出了名的资源昂贵,需要作为输入长度$L$的函数的串行时间和存储器中的$O(L^2)$。最近的工作提出了各种线性的自我注意机制,缩放串行计算仅为$O(L)$。我们进行了彻底的分析,最近的Transformer机制与线性自我关注,表演者,在整体计算复杂性。我们观察到一个显着的计算的灵活性:前向和后向传播可以进行没有近似使用次线性存储器作为$L$的函数(除了可以忽略的存储输入序列),在并行设置中的更大的时间复杂度的成本。在极端情况下,Performer在训练期间仅消耗$O(1)$内存,并且仍然需要$O(L)$时间。这种发现的时间-内存权衡可以用于训练,或者由于完全向后兼容性,可以用于低内存设备(例如智能手机或早期GPU)上的微调,从而有助于分散和民主化的深度学习。

英文摘要:

The Transformer architecture has revolutionized deep learning on sequential data, becoming ubiquitous in state-of-the-art solutions for a wide variety of applications. Yet vanilla Transformers are notoriously resource-expensive, requiring $O(L^2)$ in serial time and memory as functions of input length $L$. Recent works proposed various linear self-attention mechanisms, scaling only as $O(L)$ for serial computation. We perform a thorough analysis of recent Transformer mechanisms with linear self-attention, Performers, in terms of overall computational complexity. We observe a remarkable computational flexibility: forward and backward propagation can be performed with no approximations using sublinear memory as a function of $L$ (in addition to negligible storage for the input sequence), at a cost of greater time complexity in the parallel setting. In the extreme case, a Performer consumes only $O(1)$ memory during training, and still requires $O(L)$ time. This discovered time-memory tradeoff can be used for training or, due to complete backward-compatibility, for fine-tuning on a low-memory device, e.g. a smartphone or an earlier-generation GPU, thus contributing towards decentralized and democratized deep learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑