arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04793cs.LGcs.CL
收藏

SqueezeAttention:通过逐层最优预算实现大语言模型推理中KV缓存的二维管理

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

School of Computer Science, Peking University(北京大学计算机学院);Geoming AI(格明人工智能)

Zihao Wang, Bin Cui, Shaoduo Gan

AI总结:

针对现有KV缓存压缩算法对所有层分配相同预算的次优问题,本研究提出SqueezeAttention,从序列和层两个维度联合优化KV缓存,实现显著的内存减少与吞吐量提升。

摘要:

优化大语言模型(LLM)的键值(KV)缓存被认为是降低推理成本的关键。现有大多数KV缓存压缩算法都试图利用token重要性的差异来稀疏化token序列。然而,这些方法大多对所有层一视同仁,为每一层分配相同的KV预算。这种方式并非最优,因为某些层对输入token的敏感度可能较低,却仍获得与其他层相同的预算。在本研究中,我们发现通过识别注意力层的重要性,我们可以从序列维度和层维度这两个维度联合优化KV缓存。基于我们对推理过程中层级重要性的观察,我们提出了SqueezeAttention,可在运行时精确优化各层之间的KV缓存预算分配,随后结合三种代表性的序列维度算法,为每一层按照其专属预算压缩KV缓存。具体而言,我们首先通过计算自注意力层前后输入提示差异的余弦相似度来衡量每一层的重要性。基于该相似度,我们将层分为两组,并相应调整其KV预算。通过从序列和层两个维度优化KV缓存,SqueezeAttention在多种大语言模型和基准测试中实现了约30%至70%的内存减少,以及最高达2.2倍的吞吐量提升。代码可在https://github.com/hetailang/SqueezeAttention获取。

英文摘要:

Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. However, most of these methods treat all layers equally, allocating the same KV budget to each layer. This approach is suboptimal, as some layers may be less sensitive to input tokens yet still receive the same budget as others. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions, i.e., sequence-wise and layer-wise. Based on our observations regarding layer-wise importance in inference, we propose SqueezeAttention to precisely optimize the allocation of KV-cache budget among layers on-the-fly and then incorporate three representative sequence-wise algorithms to compress the KV-cache for each layer with its very own budget. Specifically, we first measure each layer's importance by calculating the cosine similarity of the input prompt differences before and after the self-attention layers. Based on this similarity, we then categorize the layers into two groups and adjust their KV budgets accordingly. By optimizing the KV-cache from both sequence's and layer's dimensions, SqueezeAttention achieves around 30% to 70% of the memory reductions and up to 2.2 times of throughput improvements in a wide range of LLMs and benchmarks. The code is available at https://github.com/hetailang/SqueezeAttention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑