arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07443cs.LGcs.CCcs.DScs.ITmath.IT
收藏

Attention的I/O复杂度,或者说Flash Attention有多最优?

The I/O Complexity of Attention, or How Optimal is Flash Attention?

University of California, San Diego(加利福尼亚大学圣迭戈分校)

Barna Saha, Christopher Ye

AI总结:

该研究针对FlashAttention算法的I/O复杂度最优性,证明了其在任意缓存大小下的下界匹配性,并提出了更优算法及通信复杂度与I/O复杂度的新联系。

摘要:

自注意力机制是广受欢迎的Transformer架构的核心,但受制于二次方的时间和内存复杂度。突破性的FlashAttention算法揭示了I/O复杂度是扩展Transformer的真正瓶颈。给定两层内存层次结构,即快速缓存(如GPU片上SRAM)和慢速内存(如GPU高带宽内存),I/O复杂度衡量的是对内存的访问次数。FlashAttention使用$\frac{N^2d^2}{M}$次I/O操作计算注意力,其中$N$是注意力矩阵的维度,$d$是头维度,$M$是缓存大小。然而,这种I/O复杂度是最优的吗?已知的下界仅在$M=Θ(Nd)$时排除了$o(Nd)$的I/O复杂度,因为需要写入慢速内存的输出为$Ω(Nd)$。这引出了我们工作的主要问题:FlashAttention的I/O复杂度对所有$M$值都是最优的吗?我们通过展示一个I/O复杂度下界,在任何常数因子内匹配FlashAttention为任意$M \geq d^2$值提供的上界,全面解决了上述问题。此外,我们针对$M < d^2$的情况提出了一种具有更低I/O复杂度的更优算法,并证明其同样是最优的。而且,我们的下界不依赖于使用组合矩阵乘法来计算注意力矩阵。我们证明,即使使用快速矩阵乘法,上述I/O复杂度界限也无法被改进。我们通过引入一种用于矩阵压缩的全新通信复杂度协议,并将通信复杂度与I/O复杂度联系起来实现了这一点。据我们所知,这是首个建立通信复杂度与I/O复杂度之间联系的工作,我们相信这种联系将具有独立的意义,并将在未来证明I/O复杂度下界中找到更多应用。

英文摘要:

Self-attention is at the heart of the popular Transformer architecture, yet suffers from quadratic time and memory complexity. The breakthrough FlashAttention algorithm revealed I/O complexity as the true bottleneck in scaling Transformers. Given two levels of memory hierarchy, a fast cache (e.g. GPU on-chip SRAM) and a slow memory (e.g. GPU high-bandwidth memory), the I/O complexity measures the number of accesses to memory. FlashAttention computes attention using $\frac{N^2d^2}{M}$ I/O operations where $N$ is the dimension of the attention matrix, $d$ the head-dimension and $M$ the cache size. However, is this I/O complexity optimal? The known lower bound only rules out an I/O complexity of $o(Nd)$ when $M=Θ(Nd)$, since the output that needs to be written to slow memory is $Ω(Nd)$. This leads to the main question of our work: Is FlashAttention I/O optimal for all values of $M$? We resolve the above question in its full generality by showing an I/O complexity lower bound that matches the upper bound provided by FlashAttention for any values of $M \geq d^2$ within any constant factors. Further, we give a better algorithm with lower I/O complexity for $M < d^2$, and show that it is optimal as well. Moreover, our lower bounds do not rely on using combinatorial matrix multiplication for computing the attention matrix. We show even if one uses fast matrix multiplication, the above I/O complexity bounds cannot be improved. We do so by introducing a new communication complexity protocol for matrix compression, and connecting communication complexity to I/O complexity. To the best of our knowledge, this is the first work to establish a connection between communication complexity and I/O complexity, and we believe this connection could be of independent interest and will find many more applications in proving I/O complexity lower bounds in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑