更快的邻域注意力:在线程块级别降低自注意力的O(n²)开销
Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level
AI总结:
针对邻域注意力现有内核功能与性能受限的问题,提出批量GEMM实现与融合邻域注意力两种方法,大幅提升运行速度并实现恒定内存占用。
摘要:
邻域注意力通过将每个token的注意力范围限制在其最近邻范围内,降低了自注意力的计算开销。这种由窗口大小和扩张因子参数化的限制,构成了线性投影与自注意力之间一系列可能的注意力模式。长期以来,邻域注意力以及更普遍的滑动窗口注意力模式一直受限于基础设施,尤其是在更高秩空间(二维和三维)中,因此需要开发定制内核,但现有定制内核要么功能受限,要么性能受限,甚至两者兼具。在本研究中,我们旨在通过提供两种实现邻域注意力的新方法,大幅改进现有基础设施。我们首先证明,邻域注意力可以表示为批量GEMM(通用矩阵乘法)问题,与标准注意力类似,并针对一维和二维邻域注意力实现了该内核。与现有的一维和二维邻域注意力原生CUDA内核相比,这些内核的全精度运行时间平均分别提升了895%和272%。我们发现,除了严重受限于内存带宽之外,所有非融合式邻域注意力实现都存在某些固有的低效问题,在大多数情况下,这些问题会抵消其理论上的效率增益。受融合点积注意力内核研究进展的启发,我们开发了融合邻域注意力:这是对融合点积注意力内核的改进,可对不同空间轴上的注意力进行细粒度控制。邻域注意力以将自注意力的二次时间复杂度降至线性复杂度而著称,如今它还能拥有更低的恒定内存占用,以及打破纪录的半精度运行时间。我们观察到,我们的融合实现成功规避了非融合实现中一些不可避免的低效问题……
英文摘要:
Neighborhood attention reduces the cost of self attention by restricting each token's attention span to its nearest neighbors. This restriction, parameterized by a window size and dilation factor, draws a spectrum of possible attention patterns between linear projection and self attention. Neighborhood attention, and more generally sliding window attention patterns, have long been bounded by infrastructure, particularly in higher-rank spaces (2-D and 3-D), calling for the development of custom kernels, which have been limited in either functionality, or performance, if not both. In this work, we aim to massively improve upon existing infrastructure by providing two new methods for implementing neighborhood attention. We first show that neighborhood attention can be represented as a batched GEMM problem, similar to standard attention, and implement it for 1-D and 2-D neighborhood attention. These kernels on average provide 895% and 272% improvement in full precision runtime compared to existing naive CUDA kernels for 1-D and 2-D neighborhood attention respectively. We find that aside from being heavily bound by memory bandwidth, certain inherent inefficiencies exist in all unfused implementations of neighborhood attention, which in most cases undo their theoretical efficiency gain. Motivated by the progress made into fused dot-product attention kernels, we developed fused neighborhood attention; an adaptation of fused dot-product attention kernels that allow fine-grained control over attention across different spatial axes. Known for reducing the quadratic time complexity of self attention to a linear complexity, neighborhood attention can now enjoy a reduced and constant memory footprint, and record-breaking half precision runtime. We observe that our fused implementation successfully circumvents some of the unavoidable inefficiencies in unfused implementations...
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。