arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.05682cs.LG
收藏

自注意力不需要$O(n^2)$记忆

Self-attention Does Not Need $O(n^2)$ Memory

Google Research(谷歌研究院)

Markus N. Rabe, Charles Staats

摘要:

我们提出了一个非常简单的算法注意,需要$O(1)$内存相对于序列长度和自注意力,需要$O(\log n)$内存的扩展。这与人们常说的自注意力需要$O(n^2)$记忆的观点形成了鲜明对比。虽然时间复杂度仍然是$O(n^2)$,但设备内存而不是计算能力通常是现代加速器的限制因素。因此,减少注意力的记忆要求允许处理比其他方式可行的更长的序列。我们提供了一个实际的实现加速器,需要$O(\sqrt{n})$内存,是数字稳定,是在百分之几的运行时间内的标准实现的注意力。我们还演示了如何区分功能,同时保持内存效率。对于序列长度为16384的情况,自注意力的内存开销对于推理减少了59倍,对于微分减少了32倍。

英文摘要:

We present a very simple algorithm for attention that requires $O(1)$ memory with respect to sequence length and an extension to self-attention that requires $O(\log n)$ memory. This is in contrast with the frequently stated belief that self-attention requires $O(n^2)$ memory. While the time complexity is still $O(n^2)$, device memory rather than compute capability is often the limiting factor on modern accelerators. Thus, reducing the memory requirements of attention allows processing of longer sequences than might otherwise be feasible. We provide a practical implementation for accelerators that requires $O(\sqrt{n})$ memory, is numerically stable, and is within a few percent of the runtime of the standard implementation of attention. We also demonstrate how to differentiate the function while remaining memory-efficient. For sequence length 16384, the memory overhead of self-attention is reduced by 59X for inference and by 32X for differentiation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑