Lightning Attention-2:在大语言模型中处理无限序列长度的免费午餐
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
AI总结:
本文提出Lightning Attention-2,通过分块策略分别处理块内和块间注意力计算,首次使线性注意力在因果场景中实现理论计算优势,在任意序列长度下保持恒定训练与推理速度。
摘要:
线性注意力是一种高效的注意力机制,近年来已成为传统softmax注意力的一个有前景的替代方案。凭借其以线性计算复杂度处理token的能力,线性注意力在理论上可以在不牺牲速度的情况下处理无限长度的序列,即在固定内存消耗下,对不同序列长度保持恒定的训练速度。然而,由于累积求和(cumsum)的问题,当前的线性注意力算法无法在因果场景中展现其理论优势。在本文中,我们提出了Lightning Attention-2,这是首个使线性注意力能够实现其理论计算优势的线性注意力实现。为实现这一目标,我们借鉴了分块(tiling)的思想,在线性注意力计算中分别处理块内和块间分量。具体而言,我们对块内部分采用传统的注意力计算机制,对块间部分应用线性注意力核技巧。在前向和后向过程中均采用分块技术,以充分利用GPU硬件。我们在Triton中实现了该算法,使其具备IO感知能力和硬件友好性。在不同模型规模和序列长度上进行了多种实验。无论输入序列长度如何,Lightning Attention-2都保持一致的训练和推理速度,并且显著快于其他注意力机制。源代码可在https://github.com/OpenNLPLab/lightning-attention获取。
英文摘要:
Linear attention is an efficient attention mechanism that has recently emerged as a promising alternative to conventional softmax attention. With its ability to process tokens in linear computational complexities, linear attention, in theory, can handle sequences of unlimited length without sacrificing speed, i.e., maintaining a constant training speed for various sequence lengths with a fixed memory consumption. However, due to the issue with cumulative summation (cumsum), current linear attention algorithms cannot demonstrate their theoretical advantage in a causal setting. In this paper, we present Lightning Attention-2, the first linear attention implementation that enables linear attention to realize its theoretical computational benefits. To achieve this, we leverage the thought of tiling, separately handling the intra-block and inter-block components in linear attention calculation. Specifically, we utilize the conventional attention computation mechanism for the intra-blocks and apply linear attention kernel tricks for the inter-blocks. A tiling technique is adopted through both forward and backward procedures to take full advantage of the GPU hardware. We implement our algorithm in Triton to make it IO-aware and hardware-friendly. Various experiments are conducted on different model sizes and sequence lengths. Lightning Attention-2 retains consistent training and inference speed regardless of input sequence length and is significantly faster than other attention mechanisms. The source code is available at https://github.com/OpenNLPLab/lightning-attention.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。