次线性记忆:如何让表演者成为SLiM
Sub-Linear Memory: How to Make Performers SLiM
摘要:
Transformer架构彻底改变了对序列数据的深度学习,在各种应用的最先进解决方案中无处不在。然而普通的Transformers是出了名的资源昂贵,需要作为输入长度$L$的函数的串行时间和存储器中的$O(L^2)$。最近的工作提出了各种线性的自我注意机制,缩放串行计算仅为$O(L)$。我们进行了彻底的分析,最近的Transformer机制与线性自我关注,表演者,在整体计算复杂性。我们观察到一个显着的计算的灵活性:前向和后向传播可以进行没有近似使用次线性存储器作为$L$的函数(除了可以忽略的存储输入序列),在并行设置中的更大的时间复杂度的成本。在极端情况下,Performer在训练期间仅消耗$O(1)$内存,并且仍然需要$O(L)$时间。这种发现的时间-内存权衡可以用于训练,或者由于完全向后兼容性,可以用于低内存设备(例如智能手机或早期GPU)上的微调,从而有助于分散和民主化的深度学习。
英文摘要:
The Transformer architecture has revolutionized deep learning on sequential data, becoming ubiquitous in state-of-the-art solutions for a wide variety of applications. Yet vanilla Transformers are notoriously resource-expensive, requiring $O(L^2)$ in serial time and memory as functions of input length $L$. Recent works proposed various linear self-attention mechanisms, scaling only as $O(L)$ for serial computation. We perform a thorough analysis of recent Transformer mechanisms with linear self-attention, Performers, in terms of overall computational complexity. We observe a remarkable computational flexibility: forward and backward propagation can be performed with no approximations using sublinear memory as a function of $L$ (in addition to negligible storage for the input sequence), at a cost of greater time complexity in the parallel setting. In the extreme case, a Performer consumes only $O(1)$ memory during training, and still requires $O(L)$ time. This discovered time-memory tradeoff can be used for training or, due to complete backward-compatibility, for fine-tuning on a low-memory device, e.g. a smartphone or an earlier-generation GPU, thus contributing towards decentralized and democratized deep learning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。