arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.09688cs.CL
收藏

Squeezed Attention:加速长上下文长度LLM推理

Squeezed Attention: Accelerating Long Context Length LLM Inference

UC Berkeley(加州大学伯克利分校);FuriosaAI;ICSI(国际计算机科学研究所);LBNL(劳伦斯伯克利国家实验室)

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

AI总结:

针对长上下文LLM推理中固定上下文占比高的问题,提出Squeezed Attention方法,通过离线K-means聚类和质心比较选择重要键,实现KV预算减少和推理加速,在LongBench上取得显著效果。

摘要:

新兴的大型语言模型(LLM)应用需要长输入上下文来执行文档分析和代码生成等复杂任务。对于这些长上下文长度应用,输入提示的长度对推理效率构成了重大挑战,因为推理成本随序列长度线性增加。然而,对于许多此类应用,提示中的大部分上下文在不同用户输入之间是固定的,从而提供了执行离线优化的机会,以便在收到用户输入时快速处理。我们提出Squeezed Attention来加速输入上下文大部分固定的LLM应用。我们首先利用K-means聚类离线将固定上下文的键基于语义相似性分组,并用单个质心值表示每个簇。在推理过程中,我们将用户输入的查询标记与质心进行比较,以预测固定上下文中哪些键在语义上相关,然后仅使用重要键计算精确注意力,从而减少带宽和计算成本。我们还提出了算法的分层版本,可以将注意力的复杂度从线性降低到相对于固定上下文长度的对数。我们在长上下文基准(包括LongBench)上评估了我们的方法,在LLaMA-2-7B-32K、LWM-Text-Chat-1M和Longchat-7B-v1.5-32K模型上,实现了3.1倍的KV预算减少且无明显精度损失,以及最多8倍的减少且仅有0.5点的精度差距。此外,我们实现了质心比较和稀疏FlashAttention的内核,在长上下文推理的预填充和生成阶段实现了超过4倍的加速。我们的代码可在https://github.com/SqueezeAILab/SqueezedAttention获取。

英文摘要:

Emerging Large Language Model (LLM) applications require long input context in order to perform complex tasks like document analysis and code generation. For these long context length applications, the length of the input prompt poses a significant challenge in terms of inference efficiency since the inference costs increase linearly with sequence length. However, for many of these applications, much of the context in the prompt is fixed across different user inputs, thereby providing the opportunity to perform offline optimizations in order to process user inputs quickly, as they are received. We propose Squeezed Attention to accelerate LLM applications where a large portion of the input context is fixed. We first leverage K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. During inference, we compare query tokens from the user input with the centroids to predict which keys from the fixed context are semantically relevant, and then compute exact attention using only the important keys, thereby reducing bandwidth and computational costs. We also present a hierarchical version of our algorithm which can reduce the complexity of attention from linear to logarithmic with respect to the fixed context length. We evaluate our method on long-context benchmarks including LongBench, where it achieves a 3.1$\times$ reduction in KV budget with no noticeable accuracy loss and up to an 8$\times$ reduction with only a 0.5 point accuracy gap for the LLaMA-2-7B-32K, LWM-Text-Chat-1M, and Longchat-7B-v1.5-32K models. Futhermore, we implement kernels for centroid comparison and sparse FlashAttention with important keys, achieving more than 4$\times$ speedups during both the prefill and generation phases for long-context inference. Our code is available at https://github.com/SqueezeAILab/SqueezedAttention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑